NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY,  CALIFORNIA 


THESIS 


DEVELOPING  BEHAVIORAL  METRICS  FOR 
DECISION-MAKING  IN  MARINE  CORPS  SMALL-UNITS 

by 

Jonathon  J.  Richardson 
September  2013 

Thesis  Advisor:  Susan  Hutchins 

Thesis  Co-Advisor:  Lee  Sciarini 

Second  Reader:  Quinn  Kennedy 


This  thesis  was  performed  at  the  MOVES  Institute 
Approved  for  public  release;  distribution  is  unlimited 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


REPORT  DOCUMENTATION  PAGE  |  Form  Approved  OMB  No  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instruction, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send 
comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to 
Washington  headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington, 
VA22202^1302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188)  WashingtonDC20503. 

I.  AGENCY  USE  ONLY  ( Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

September  2013  Master’ s  Thesis 

4.  TITLE  AND  SUBTITLE  5.  FUNDING  NUMBERS  — — 

DEVELOPING  BEHAVIORAL  METRICS  FOR  DECISION-MAKING  IN 
MARINE  CORPS  SMALL-UNITS _ 

6.  AUTIIOR(S)  Jonathon  .1.  Richardson 

7.  PEREORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(F.S)  8.  PERFORMING  ORGANIZATION 

Naval  Postgraduate  School  REPORT  NUMBER 

Monterey,  CA93943-5000 

9.  SPONSORING  /MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES)  10.  SPONSORING/MONITORING 
N/A  AGENCY  REPORT  NUMBER 

II.  SUPPLEMENTARY  NOTES  The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official  policy 
or  position  of  the  Department  of  Defense  or  the  U.S.  Government.  IRB  Protocol  number  NPS.2013.0041-IR-EP7-A 

12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT  12b.  DISTRIBUTION  CODE  ” 

Approved  for  public  release;  distribution  is  unlimited 

This  thesis  developed  behaviorally  anchored-rating  scales  (BARS)  for  use  in  evaluating  Marine  Corps  small-units 
(SUs)  during  live  and  virtual  decision-making  (DM)  training.  Currently,  the  Marine  Corps  does  not  mandate  the  use 
of  standardized  metrics  and  processes  for  the  evaluation  and  feedback  during  SU  DM  training  scenarios.  Often, 
evaluators  assess  trainee  performance  in  uncertain  situations  based  on  gross  outcomes  rather  than  process-oriented 
measures.  Properly  developed  and  integrated  into  a  training  plan,  BARS  offers  a  novel  solution  to  these  constraints. 
The  application  of  BARS  has  been  explored  in  the  law  enforcement  and  medical  domains  and  has  been  explored  on  a 
limited  basis  for  infantry  tasks.  This  effort  proposed  that  observing  SU  leader’s  observable  behaviors  in  context  of 
their  task  performance  would  provide  objective  measures  of  DM.  The  first  portion  of  this  work  was  focused  on  the 
design  and  development  of  BARS  specific  for  use  during  multiple  IIT  scenarios.  The  second  phase  of  this  research 
refined  the  initial  BARS  into  six  scenario-specific  evaluation  measurement  tools.  Additionally,  this  work  provided  an 
initial  effectiveness  evaluation  of  the  refined  BARS  in  operational  training  at  the  IIT.  Results  indicate  that  the  BARS 
developed  for  this  thesis  have  the  potential  to  replace  the  nonstandard  and  subjective  methods  currently  used  for  the 
evaluation  of  DM  training  in  live  and  virtual  environments. 


14.  SUBJECT  TERMS  Decision-Making,  Metrics,  Behaviorally-Anchored  Rating  Scales,  BARS,  I  15.  NUMBER  OF 


Cognitive  Task  Analysis 

PAGES 

123 

16.  PRICE  CODE 

17.  SECURITY 

18.  SECURITY 

19.  SECURITY 

20.  LIMITATION  OF 

CLASSIFICATION  OF 

CLASSIFICATION  OF  THIS 

CLASSIFICATION  OF 

ABSTRACT 

REPORT 

PAGE 

ABSTRACT 

Unclassified 

Unclassified 

Unclassified 

UU 

NSN  7540-01-280-5500 

Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  239-18 


1 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


11 


Approved  for  public  release;  distribution  is  unlimited 


DEVELOPING  BEHAVIORAL  METRICS  FOR  DECISION-MAKING  IN 
MARINE  CORPS  SMALL-UNITS 


Jonathon  J.  Richardson 
Captain,  United  States  Marine  Corps 
B.S.,  ITT  Technical  Institute  of  Seattle,  2003 


Submitted  in  partial  fulfillment  of  the 
requirements  for  the  degree  of 


MASTER  OF  SCIENCE  IN 

MODELING,  VIRTUAL  ENVIRONMENTS  AND  SIMULATION 

from  the 


NAVAL  POSTGRADUATESCHOOL 
September  2013 

Author;  Jonathon  J.  Richardson 

Approved  by;  Susan  Hutchins 

Thesis  Advisor 


Lee  Sciarini 
Thesis  Co-Advisor 


Quinn  Kennedy 
Second  Reader 


Christian  Darken 

Chair,  MOVES  Academic  Committee 


Peter  Denning 

Chair,  Department  of  Computer  Science 
iii 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


IV 


ABSTRACT 


This  thesis  developed  behaviorally  anchored-rating  scales  (BARS)  for  use  in  evaluating 
Marine  Corps  small-units  (SUs)  during  live  and  virtual  decision-making  (DM)  training. 
Currently,  the  Marine  Corps  does  not  mandate  the  use  of  standardized  metrics  and 
processes  for  the  evaluation  and  feedback  during  SU  DM  training  scenarios.  Often, 
evaluators  assess  trainee  perfonnance  in  uncertain  situations  based  on  gross  outcomes 
rather  than  process-oriented  measures.  Properly  developed  and  integrated  into  a  training 
plan,  BARS  offers  a  novel  solution  to  these  constraints.  The  application  of  BARS  has 
been  explored  in  the  law  enforcement  and  medical  domains  and  has  been  explored  on  a 
limited  basis  for  infantry  tasks.  This  effort  proposed  that  observing  SU  leader’s 
observable  behaviors  in  context  of  their  task  perfonnance  would  provide  objective 
measures  of  DM.  The  first  portion  of  this  work  was  focused  on  the  design  and 
development  of  BARS  specific  for  use  during  multiple  IIT  scenarios.  The  second  phase 
of  this  research  refined  the  initial  BARS  into  six  scenario-specific  evaluation 
measurement  tools.  Additionally,  this  work  provided  an  initial  effectiveness  evaluation  of 
the  refined  BARS  in  operational  training  at  the  IIT.  Results  indicate  that  the  BARS 
developed  for  this  thesis  have  the  potential  to  replace  the  nonstandard  and  subjective 
methods  currently  used  for  the  evaluation  of  DM  training  in  live  and  virtual 
environments. 
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I.  INTRODUCTION 


The  United  States  Marine  Corps  (USMC)  has  been  fighting  and  defeating  our 
nation’s  enemies  for  237  years.  Regardless  of  the  clime  or  place,  the  Marine  Air-Ground 
Task  Force  (MAGTF)  places  ever-increasing  responsibility  on  its  Marines,  particularly  at 
the  non-commissioned  and  company  officer  grades  in  order  to  remain  adaptable  to  any 
environment  (Hannigan  et  ah,  2012).  The  concept  of  the  “Strategic  Corporal,”  a  term 
coined  by  Gen  Charles  C.  Krulak,  emphasizes  that  actions  made  by  individual  Marines  at 
the  tactical  level  can  influence  the  operational  and  strategic  levels  of  warfare  (Krulak, 
1999).  This  type  of  individual  decision-making  at  the  tactical  level  makes  educating  and 
training  marines  and  sailors  in  ethics  and  decision-making  in  complex  and  uncertain 
environments  imperative  (Kobus,  2012).  Concomitantly,  a  means  to  measure  and 
evaluate  a  Marine’s  or  Sailor’s  decision-making  and  leadership  ability  in  such 
environments  is  critical  to  enabling  the  future  success  of  the  MAGTF  (Dunford,  2011). 

The  ambiguous  and  complex  nature  of  modern  warfare  puts  a  premium  on  small- 
unit  leadership  at  the  squad,  platoon,  and  company  organizational  levels  (Gideons,  2011). 
Leaders  reacting  to  a  dynamic  and  chaotic  battlespace,  which  is  compounded  by 
geographic  dispersion,  mission  diversity,  adaptive  adversaries,  and  rules  of  engagement 
(ROE)  (Board,  2012),  are  required  to  make  intelligent,  timely,  and  situationally  relevant 
decisions.  Additionally,  deficiencies  in  time  and  information  further  complicate  a 
leader’s  ability  to  fully  process  and  analyze  the  myriad  of  possibilities. 

On  an  important  decision  one  rarely  has  100%  of  the  information  needed 
for  a  good  decision  no  matter  how  much  one  spends  or  how  long  one 
waits.  And,  if  one  waits  too  long,  he  has  a  different  problem  and  has  to 
start  all  over.  This  is  the  terrible  dilemma  of  the  hesitant  decision  maker. 
(Greenleaf  &  Spears,  2002,  p.  36) 

Marines  and  sailors  need  to  be  able  to  quickly  and  accurately  make  decisions  in 
combat,  despite  the  complexities,  and  instructors  and  supervising  officers  need  tools  to 
help  them  evaluate  the  trainees’  ability  to  do  so.  The  art  and  science  of  decision-making 
encompasses  more  than  a  decision  maker’s  speedy  computational  prowess.  Decision¬ 
making  requires  situational  awareness  of  the  problem  at  hand,  along  with  the  creative 
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means,  informed  by  experience,  education  and  intelligence,  to  achieve  the  desired 
solution  (USMC,  1997). 

Colonel  John  Boyd’s  (USAF)  “Observe,  Orient,  Decide  and  Act”  (OODA)  loop  is 
an  abstraction  of  the  decision-making  cycle  that  is  reliant  upon  an  individual  first 
observing  the  environmental  conditions  or  cues.  Once  the  individual  has  detected  the 
cues  in  the  environment,  the  Marine  or  Sailor  must  adjust  his  or  her  mental  picture  to  the 
situational  reality  and  then  decide  on  a  course  of  action  or  inaction  (Angerman,  2004). 
This  decision-making  cycle  is  a  descriptive  framework  that  military  leaders  and 
organizations  use  to  understand  and  analyze  the  human  command  and  control  (C2) 
system  (Hannigan  et  ah,  2012). 

Decision-making  theories  and  models  are  categorized  as  involving  either 
analytical  or  intuitive  strategies  (Sjoberg,  2003).  An  example  of  an  analytical  strategy  is 
found  in  the  Vroom-Jago  “Rational  Decision”  model  (Turpin  &  Marais,  2004).  Gary 
Klein’s  “Recognition  Primed  Decision”  (RPD)  model  and  Boyd’s  “OODA”  theory  are 
categorized  as  intuitive  strategy  models  (Turpin  &  Marais,  2004).  Analytical  decision 
models  are  more  appropriate  for  simple  scenarios  where  time  is  sufficient  to  evaluate 
alternatives  for  a  relatively  few  courses  of  action  (COAs)  and  the  decision  maker  has 
access  to  the  information  needed  to  make  a  decision.  Intuitive  decision  models  are  more 
appropriate  for  complex  or  uncertain  scenarios  where  deficiencies  in  time,  information 
and/or  resources  will  not  impede  the  selection  of  a  sufficient  COA  (Roth,  2004). 
Additionally,  intuitive  decision-making  strategies  take  into  consideration  the  operational 
factors  experienced  by  deployed  Marines  and  Sailors  (Commandant  of  the  Marine  Corps, 
201 0B).  Due  to  these  factors,  this  research  will  focus  on  the  intuitive  strategies  instead  of 
analytical  methods  when  discussing  the  decision-making  process. 

The  Commandant  of  the  Marine  Corps  (CMC),  Gen.  James  F.  Amos,  established 

a  set  of  priorities  for  the  current  and  future  outlook  of  the  USMC.  His  third  doctrinal 

priority  is,  paraphrasing,  to  improve  training  and  experience  levels  for  maneuver  unit 

squad  leaders  (Amos,  2010).  Additionally,  the  CMC  specifically  points  out  that  the 

Naval  Postgraduate  School  and  marines  attending  the  institution  direct  their  individual 

studies  to  “ensure  they  align  with  USMC  needs”  (Amos,  2010,  p.  14).  Science  and 
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Technology  Objectives  (STOs)  codify  these  identified  gaps  and  address  them  under  the 
purview  of  the  Office  of  Naval  Research  (ONR).  Research  conducted  for  this  thesis  will 
support  the  Human  Performance,  Training  and  Education  (HPT&E)  thrust,  aligning  with 
the  Test  and  Evaluation  (T&E)  STO  #2,  “Small  unit  learning  and  performance 
assessment”  (ONR,  2012). 

Current  operational  training  of  Marine  Company,  Platoon,  and  Squad-based  units 
is  done  on  training  ranges  in  Live,  Virtual  and/or  Constructive  (LVC)  environments.  The 
Infantry  Immersion  Trainer  (IIT)  located  on  Marine  Corps  Base  Camp  Pendleton 
(MCBCP),  CA,  is  an  example  of  a  live  and  virtual  operational  training  environment 
(Kobus,  2012).  For  this  thesis  research,  the  IIT  on  MCBCP  served  as  the  facility  for 
observational  data  collection.  Currently,  IIT  staff  conducts  subjective  performance 
assessments,  usually  in  the  form  of  After  Action  Reviews  (AARs).  These  AARs 
traditionally  occur  at  the  conclusion  of  a  scenario  or  on  an  ad  hoc  basis  at  the  discretion 
of  the  training  unit  (Kobus,  2012).  While  these  subject  matter  experts  (SMEs)  provide 
critical  expertise  and  insight,  feedback  tends  to  be  limited  (Kobus,  2012).  Moreover, 
there  are  neither  standardized  metrics  for  evaluating  the  small  unit’s  behavior  while 
engaging  in  scenarios,  nor  standardized  processes  for  providing  feedback  (Kobus,  2012). 
Including  behaviorally-anchored  metrics  in  the  training  and  evaluation  process  at  the  IIT 
will  improve  feedback  to  trainees.  Measuring  the  small-unit’s  perfonnance  in  this 
context  will  improve  assessments  of  decision-making  in  certain  IIT  scenarios,  thus 
contributing  to  the  standardization  of  Marine  operational  training  (Kobus,  2012). 

This  thesis  research  is  segmented  into  two  phases,  the  knowledge  elicitation  phase 
and  the  observational  data-collection  phase.  For  the  knowledge  elicitation  phase  (phase 
one),  semi-structured  interviews  were  perfonned  with  SMEs  to  elicit  information  relevant 
to  developing  measures  to  assess  performance.  These  SME  interviews  were  used  to 
develop  a  set  of  observer-based  performance  measures  used  in  specific  IIT  related 
scenarios  as  a  proof  of  concept. 

The  output  from  the  knowledge  elicitation  phase  (phase  one)  was  used  to  develop 

a  set  of  behaviorally-anchored  metrics  for  use  in  evaluating  the  performance  of  Marine 

Corps  squads  engaging  in  training  at  the  IIT.  These  metrics  were  used  by  the  researcher 
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in  the  observational  data  collection  phase  to  collect  performance  data  at  the  IIT.  The  goal 
for  this  phase  was  to  test  the  metrics  developed  in  the  first  phase  to  determine  if  they 
provide  objective  and  effective  measures.  Developing  valid  and  reliable  metrics  will 
facilitate  assessment  of  individual  and  collective  performance  and  overall  training 
effectiveness.  “These  measures  will  enhance  and  complement  current  subjective 
measures”  (Kobus,  2012,  p.  28). 

Live  and  virtual-training  environments  such  as  the  IIT  provide  a  new  approach  to 
instruct,  train  and  evaluate  marines  and  sailors.  Compared  to  traditional  Military 
Operations  on  Urban  Terrain  (MOUT)  facilities  and  ranges,  these  new  training  ranges 
offer  more  dynamic  and  immersive  experiences  approximating  realistic  combat 
environments  (United  States  Joint  Forces  Command,  2011).  The  objective  for  this  thesis 
research  was  to  provide  the  evaluator  a  means  to  quantify  decision-making  performance. 
The  benefit  from  this  research  is  that  it  provides  instructors  behaviorally-anchored 
measures  that  can  be  used  to  provide  specific  relevant  feedback  to  trainees.  Through 
realistic  training,  Marines  will  be  prepared  for  complex  and  ambiguous  situations  by 
leveraging  proficiency  in  technical  and  tactical  knowledge,  while  executing  sound 
judgment  in  a  decentralized  manner  (Conway,  2008). 
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II.  BACKGROUND 


A.  HISTORY  OF  ASSESSMENTS 

The  sinking  of  the  British  passenger  liner  RMS  Lusitania  by  a  Gennan  submarine 
in  1915  (Lauriat,  1915),  and  the  unrestricted  German  submarine  campaign  of  1917 
(Tucker  &  Roberts,  2005)  thrust  the  United  States  into  the  “War  to  End  all  Wars”  of 
WWI  (Jamieson,  1988,  p.  101).  Americans  such  as  Robert  M.  Yerkes,  president  of  the 
American  Psychological  Association  (APA),  were  eager  to  aid  the  war  effort.  He  quickly 
convened  the  APA  in  an  effort  to  pool  their  collective  knowledge,  methods,  and  expertise 
in  order  to  increase  the  efficiency  and  effectiveness  of  screening  and  assigning  enlistees 
and  officers  to  military  positions  (Kevles,  1968).  Along  with  noted  psychologists,  Robert 
L.  Thorndike,  Lewis  Tennan,  and  David  Wechsler,  the  APA  created  an  alpha  and  beta 
assessment  battery  that  would  evaluate  and  then  classify  the  intellectual  potential  of 
military  enlistees  and  officer  candidates  (Kevles,  1968;  Cardona  &  Elspeth,  2007).  By 
1976,  these  initial  assessment  tools  were  refined,  and  then  formally  presented  as  the 
Anned  Services  Vocational  Aptitude  Battery  (ASVAB).  Currently,  the  Department  of 
Defense  (DoD)  uses  “this  test  battery  for  both  screening  qualified  candidates  and 
assessing  trainability  for  classified  jobs”  (Dubay,  2007,  p.  14). 

An  assessment  “determine[s]  the  importance,  size,  or  value  of’  a  process  by  an 
individual  or  organization  (“Assess,”  n.  d.).  In  traditional  educational  curriculums, 
assessments  are  characterized  as  having  either  a  formative  or  summative  format.  The 
intent  of  both  fonnats  is  to  gauge  the  learner’s  comprehension  of  the  curriculum  and 
validate  the  instructor(s)  or  institution’s  method  of  delivering  the  education  or  training 
(Wiliam,  2000).  The  summative  format  is  a  judgment  at  a  milestone  in  the  learner’s 
knowledge  acquisition  process  (Taras,  2005).  The  formative  assessment  entails  an 
iterative  loop  that  adjusts  and  corrects  a  learner’s  gap  or  deviation  in  knowledge  from  an 
established  knowledge  standard  (Taras,  2005).  The  intent  of  either  assessment  format  is 
accountability  to  its  internal  (teachers,  students,  and  parents)  and  external  stakeholders 
(school  boards,  government,  etc.)  (Linn,  2000;  Shepard,  2000A). 
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Historically,  assessments  have  been  used  to  refonn  a  particular  social  policy,  such 
as  Head  Start,  after-school  programs,  school  uniforms,  or  instructional  strategy  (Linn, 
2000).  The  physical  implementation  of  assessments  is  typically  relatively  inexpensive, 
transparent,  and  easily  implementable  which  provides  a  smooth  promotion  and  inclusion 
into  educational  enviromnents  (Linn,  2000).  Figure  1  illustrates  the  evolving  nature  and 
application  of  assessments  with  regard  to  curriculum,  function,  and  learning  theory. 


Reformed  Vision 
of  Curriculum 


/  \  \  / \ 
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(circa  1980s  -  2000+) 


Emergent  Paradigm 
(circa  1990s  -  2000+) 


Figure  1.  The  historical  and  evolving  viewpoints  of  assessment,  curriculum,  and 

learning  theory  (From  Shepard,  2000B,  p.  4). 


Twentieth  century  approaches  to  assessment  viewed  instruction  as  a  process  to 
eliminate  inefficiencies  from  a  particular  learner’s  behavior  (Shepard,  2000B).  Under 
this  paradigm,  learners  were  expected  to  master  all  the  basic  domain  knowledge  through 
repetition  exercises  and  thereby  incrementally  develop  expertise  (Shepard,  2000A).  This 
linear  approach  to  instructional  assessment  migrated  into  summative  testing 
methodologies  such  as  the  Ohio  Graduation  Test  (OGT)  and  the  Scholastic  Assessment 
Test  (SAT)  (List  of  Standardized  Tests  in  the  United  States,  n.  d.). 

The  instructional  assessment  domain  is  currently  undergoing  a  transfonnation  in 

response  to  the  policies  in  the  2001”No  Child  left  behind  Act”  (Kim  &  Sunderman,  2005; 

Linn,  Baker,  &  Betebenner,  2002;  No  Child  Left  Behind  Act  of  2001  (2002)). 

Additionally,  the  congressionally  mandated  National  Assessment  of  Educational  Progress 

(NAEP)  instituted  a  national  standardization  protocol  that  enforces  state  education 
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performance  requirements  for  pupils  in  primary  and  secondary  schools  (Swanson  & 
Stevenson,  2002).  These  rules  are  based  upon  egalitarian  principles  that  are  agnostic  to 
geographic,  socioeconomic,  and/or  linguistic  differences  between  diverse  populations 
(Kim  &  Sunderman,  2005).  Instituting  a  carrot-and-stick  approach  to  refonning 
education  and  assessment  means  educators,  pupils,  and  schools  are  being  held 
accountable  for  the  educational  outcome  of  the  pupil.  Therefore,  primary  and  secondary 
education  will  ideally  provide  pupils  “a  fair,  equal,  and  significant  opportunity  to  obtain  a 
high-quality  education  and  reach,  at  a  minimum,  proficiency  on  challenging  state 
academic  achievement  standards  and  state  academic  assessments”  (NCLB,  2001,  Sec. 
1001  [3]). 

The  domain  of  learning  theory,  and  more  specifically  knowledge  development,  is 
evolving  with  advances  in  medicine,  science,  and  technology  (Dede  &  Barab,  2009; 
Osborne  &  Wittrock,  1983).  Traditional  associative  and  behaviorist  approaches  to 
learning  and  instruction  are  giving  way  to  newer  mental  models  based  on  cognitive  and 
constructivist  research  and  their  applied  learning  theories  (Anderson,  Reder,  Simon, 
Ericsson,  &  Glaser,  1998).  These  emergent  approaches  are  less  concerned  with  solely 
eliminating  inefficiencies  in  the  learner’s  behavior.  Cognitive  and  constructivist 
approaches  focus  on  tailoring  knowledge  development  to  the  individual’s  social  and 
cultural  learning  biases  and  aptitudes  (Shepard,  2000B).  This  movement  in  learning 
science  has  promoted  the  integration  of  formative  assessment  methodologies  in  addition 
to,  or  in  place  of,  traditional  summative  regimes  (Harlen  &  James,  1997;  Shepard, 
2000A;  Shepard,  2000B). 

B.  HISTORY  OF  THE  AFTER  ACTION  REVIEW 

The  most  valuable  and  impactful  assessment  used  in  operational  training  is  the  After 

Action  Review,  or  AAR  (Smith,  Yates,  Valdyke,  Roby,  &  Denney,  2010).  The  AAR,  as  a 

training  tool,  came  into  existence  approximately  45  years  ago  and  was  predicated  upon  two 

important  influences  (Morrison  &  Meliza,  1999).  The  first  influence  was  S.  L.  A.  Marshall’s, 

“Island  Victory:  The  Battle  of  Kwajalein  Atoll.”  In  this  book,  he  employs  “the  interview  after 

combat”  method,  where  he  recalls  the  “very  words  of  the  men  who  fought  -  and  therefore  a 

highly  accurate  account  of  exactly  what  happened,  as  one  scene  of  stress  and  confusion 
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followed  another”  (Marshall  &  Dawson,  1944,  p.  xxi;  cited  in  Morrison  &  Meliza,  1999).  The 
other  influence  was  the  “performance  critique”  method  (Morrison  &  Meliza,  1999).  This  was 
the  primary  means  of  delivering  perfonnance  feedback  prior  to  the  development  of  the  AAR. 
This  was  originally  employed  to  detenmine  battle  damage  assessments  during  simulated  battles 
where  the  outcomes  of  the  exercises  were  subjectively  adjudicated  by  “human  umpires” 
(Morrison  &  Meliza,  1999,  p.  6).  In  the  early  1970s,  the  perfonnance  critique  method  was 
detennined  to  be  an  ineffective  means  to  assess  training  units  and  enhance  their  perfonnance 
goals.  From  this  coalescence,  the  DoD  developed  the  AAR  methodology  to  provide  feedback 
in  a  more  constructive  and  non-punitive  manner  (Monison  &  Meliza,  1999). 

C.  AAR  AND  MILITARY  TRAINING  APPLICATIONS 

In  the  military  context,  formative  assessments  are  congruent  with  the  AAR  format 
(Morrison  &  Meliza,  1999;  Rudolph,  Simon,  Raemer  &  Eppich,  2008).  The  AAR  and  its 
feedback  mechanism  afford  individual  and  group  learners  a  means  to  correct  deviations 
in  either  an  informal  or  instrumented  (formal)  manner  (Kobus,  2012).  In  either  AAR 
format,  there  must  be  a  blending  of  training  and  assessment  feedback  that  the  instructor 
must  weave  into  a  narrative  that  the  learner  understands  and  then  is  responsive  toward 
(United  States  Marine  Corps  Training  and  Education  Command  (TECOM),  2011). 
“Education’  teaches  [the  learner]  how  to  think,  perceive,  and  make  decisions  while 
‘training’  teaches  what  to  think  and  do  to  make  decisions”  (TECOM,  2011,  p.  18).  The 
AAR  must  not  be  the  means  to  solve  the  learner’s  “problem”  or  train  to  the  explicit 
solution.  The  fonnative  assessment  fonnat  must  allow  the  learner  to  perceive, 
comprehend,  and  then  innovate  beyond  their  friction  and  allow  a  suitable  range  of 
alternative  actions,  instead  of  a  single  “correct”  decision  (Kobus,  2012,  p.  26-27). 

In  the  AAR,  the  trainer  must  balance  the  desire  to  explicitly  correct  deviations  or 

errors,  thus  circumventing  the  decision-making  process  of  the  learner.  Instead,  the  trainer 

should  ignore  minor  deviations  that  do  not  impact  the  final  decision  process  and  delay  the 

expected  errors  that  the  learner  has  consistently  made  by  offering  timely,  open-ended 

hints  or  leading  questions.  Only  when  delaying  expected  errors  fail  should  the  instructor 

intercede  with  a  prompt  that  initiates  the  learner  into  self-assessment  and  correction. 

Depending  on  the  constraints  of  the  exercise,  the  trainer  could  engage  the  learner  more 
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directly  with  explicit  feedback  on  performance.  This  tactical  change  of  explicitly 
interceding  and  guiding  the  training  unit  to  a  desirable  end  state  would  normally  be 
driven  by  external  real-world  pressures  (Lepper  &  Woolverton,  2002;  Shepard,  2000B). 

1.  ASSESSMENTS  ON  MILITARY  TRAINING  RANGES 

Currently,  the  DoD  uses  the  AAR  to  evaluate  trainee  performance  involving 
military  training  ranges  (Frank  et  ah,  2004).  Operational  training  in  LVC  environments 
is  advantageous  to  training  conducted  in  expeditionary  environments  because  of  the 
prohibitive  expense,  and  unmitigated  risk  to  personnel,  materiel,  and/or  resources  (Ford 
et  al.,  2004;  Boese,  2013).  The  USMC  executes  the  majority  of  its  operational  training 
on  air,  naval,  and/or  ground  training  ranges  throughout  the  continental  United  States 
(CONUS)  and  territories  (Under  Secretary  of  Defense  (Personnel  and  Readiness),  2012). 
Additionally,  selected  training  ranges  in  friendly  foreign  host  nations  are  utilized  to 
promote  friendly  military  relations  and  enhance  coalition  training  capabilities  (Sanders, 
1998).  Military  use  of  assessments  and  the  operational  training  conducted  on  various 
training  ranges  share  a  common  functional  and  mutually  dependent  end  state  (Lanman, 
Becker,  &  Samper,  2009). 

a.  USMC  Training  Ranges 

The  current  range  infrastructure  utilized  by  the  USMC  is  characterized  by 
dispersed  geographic  locations,  dissimilar  capabilities  and  capacities  with  the 
requirement  to  support  a  broad  spectrum  of  mission  essential  warfighter  training 
(TECOM,  2006).  Training  ranges  are  classified  by  their  particular  size  and/or  level  of  the 
training  unit  (smallest  to  largest);  specifically,  individual  level,  unit  level,  Marine  Air 
Ground  Task  Force  (MAGTF)  and  Marine  Expeditionary  Unit  (MEU)  Level,  and 
MAGTF  Marine  Expeditionary  Battalion  (MEB)  Level  (TECOM,  2006,  p.  II).  Their 
central  purpose  and  function  is  to  train  Marines  collectively  in  the  approximated 
operational  environment  that  enables  effective  operations  in  complex  and  uncertain 
expeditionary  environments  (Conway,  2008).  Future  reference  to  Marine  Corps  training 
ranges  will  be  specific  to  air/ground  training  ranges  located  in  CONUS  and  Hawaii. 
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b.  Infantry  Immersion  Trainer 

The  original  infantry  immersion  trainer  (IIT)  is  located  on  training  range 
“Area  62”  on  Marine  Corps  Base  Camp  Pendleton  (MCBCP)  (Babb,  2007).  The  IIT  was 
conceptualized  by  then  I  Marine  Expeditionary  Force  (I  MEF)  Commanding  General 
James  N.  Mattis.  He  wanted  and  demanded  industry,  governmental  agencies,  and 
academia  to  construct  a  training  environment  that  would  prepare  Marines  for 
deployments  to  Iraq  in  support  of  Operation  Iraqi  Freedom  (OIF).  His  training  intent  for 
the  IIT  was  for  Marines  to  be  immersed  with  the  “sights,  sounds,  smells  and  chaos  of 
urban  and  close  quarters  battle  so  that  their  first  real  firelights  were  no  worse  than  their 
simulations”  (Muller,  Schmorrow  &  Buscemi,  2008,  p.  14). 

In  2007,  with  support  from  the  Office  of  Naval  Research  (ONR),  Marine 
Corps  Warfighting  Laboratory  (MCWL),  and  I  MEF  staff,  the  IIT  concept  became  an 
operational  reality  (Muller  et  ah,  2008;  Schwetje,  2009).  The  central  mission  of  the  IIT  is 
to  “provide  the  most  realistic  combat  conditions  and  settings  to  the  small  unit,  in  a  virtual 
format,  in  order  to  teach  and  reinforce  combat  decision-making  and  small  unit  mission 
rehearsal”  (Babb,  2007).  Currently,  there  are  two  additional  IIT  facilities  located  on 
MCB  Camp  Lejeune,  NC,  and  MCB  Kaneohe  Bay,  HI.  Training  operations  commenced 
at  these  installations  on  July  and  October,  2011,  respectively.  Both  facilities  operate 
independently  of  the  other  IITs  with  regard  to  their  training  concept  of  operations 
(Kobus,  2012).  All  future  reference  to  the  IIT  will  be  specific  to  the  MCBCP  IIT. 

c.  Assessments  at  the  Infantry  Immersion  Trainer 

Currently,  assessments  of  unit  performance  during  operational  training 
exercises,  particularly  at  the  IIT,  tend  to  be  subjective  (Kobus,  2012;  Matthews  &  Beal, 
2002).  Moreover,  there  are  neither  standardized  metrics  (tools  or  applications)  for 
evaluating  the  small  unit’s  perfonnance  while  engaging  in  scenarios  nor  standardized 
processes  for  providing  objective  feedback  (Hannigan  et  ah,  2012;  Kobus,  2012). 
Additionally,  IIT  trainers  conduct  performance  assessments  on  an  ad  hoc  basis.  While 
these  subject  matter  experts  (SMEs)  provide  critical  expertise  and  insight,  feedback  tends 
to  be  limited  in  nature.  The  trainers  debriefing  narrative  identifies  “what  happened 
during  the  scenario,  why  it  happened,  and  how  it  should  be  done  in  similar  situations” 
(Kobus,  2012,  p.  24). 
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IIT  trainers  and  staff  utilize  different  assessment  techniques  for  the  pre¬ 
training,  training,  and  post-training  exercise  phases.  In  the  pre-training  phase  (prior  to 
training  execution),  situational  judgment  tests  (SJTs)  are  used  to  assess  trainee  courses  of 
actions  (COAs)  via  scenario-based  problems  to  resolve  training  effectiveness  issues 
(Kobus  &  Viklund,  2012).  Training  effectiveness  issues  pertain  to  “decision-making, 
tacit  knowledge  and  practical  ‘know-how’  domains”  (Kobus,  2012,  p.  28).  During  the 
training  phase,  trainers  provide  “as  needed”  perfonnance  assessments  via  declarative 
verbal  statements.  This  feedback  is  provided  by  an  individual  lead  trainer  or  through 
multiple  trainers  working  in  teams  (2-3  trainers).  Prompts  providing  guidance  are 
normally  provided  while  walking  alongside  the  training  unit  (Kobus,  2012).  During  the 
post- training  phase,  formative  assessments  are  conducted  in  one  of  four  AAR  formats: 
Hot  Wash,  Informal  AAR,  Contour  Camera  instrumented  AAR,  or  tactical  video¬ 
capturing  system  (TVCS)/Video  Flashlight  instrumented  AAR  (Kobus,  2012;  “Contour 
(camera  system),”  n.  d.).  These  formats  are  described  in  the  next  section. 

d.  After  Action  Review  at  the  Infantry  Immersion  Trainer 

IIT  trainers  use  four  AAR  formats  to  assess  and  debrief  the  training  units’ 
performance.  The  least  formal,  the  Hot  Wash,  is  generally  performed  when  a  more 
structured  method  is  not  required,  usually  with  unit  members  forming  a  circle  formed 
around  the  trainer.  The  trainer  briefly  provides  his  evaluation  regarding  the  unit’s 
performance  by  highlighting  his  positive  and  negative  observations  of  the  exercise.  This 
is  followed  by  the  trainer  explaining  corrective  actions  for  observed  faults  regarding  the 
unit’s  tactics,  techniques,  and  procedures  (TTPs).  This  type  of  AAR  usually  takes  no 
longer  than  10  minutes  (Kobus,  2012). 

The  informal  AAR  extends  the  Hot  Wash  with  a  few  notable  additions. 
This  method  usually  employs  a  3-D  block  model  to  visually  depict  the  eagle’s  eye  view 
of  the  training  area.  The  alternative  to  the  3-D  block  model  is  a  projection  of  the  training 
area  on  a  television  screen  or  computer  monitor  (Kobus,  2012).  The  training  unit 
members,  particularly  the  individual  squad  and/or  platoon  members,  are  the  focus  of  this 
type  of  AAR.  The  trainer  poses  probing  questions  to  elicit  feedback  from  all  members  of 
the  unit  in  order  to  share  and  hear  all  relevant  perspectives  pertaining  to  critical  errors 
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observed  in  the  exercise.  The  goal  of  having  each  member  share  their  perspective  is  to 
openly  discuss  their  thought  processes  and  decision-making  actions  (Department  of  the 
Army  (DoA),  1993;  Kobus,  2012).  This  type  of  AAR  usually  requires  no  longer  than  20 
minutes  (Kobus,  2012). 

The  next  two  types  of  AARs,  the  Contour  Camera  instrumented  AAR  and 
TVCS/Video  Flashlight  instrumented  AAR  only  differ  regarding  the  way  they  collect 
video  and  audio,  the  system’s  processing  time,  and  viewing  perspective  (“Contour 
(camera  system),”  n.  d.;  Kobus,  2012).  The  TVCS/Video  Flashlight  and  Contour  Camera 
instrumented  AARs  are  both  conducted  in  the  same  structured  classroom  environment 
(Kobus,  2012).  The  Video  Flashlight  systems  and  TVCS  are  specifically  used  during 
phase  I  and  phase  II,  respectively  (Kobus,  2012).  The  Contour  Camera  approach  can  be 
used  in  either  phase  I  or  phase  II.  In  the  TVCS/Video  Flashlight  method,  independent 
camera  operators  and  video  technicians  use  a  closed-circuit  television  (CCTV)  computer 
network  to  capture  video  streams  from  myriad  cameras  throughout  the  phase  I/II  training 
areas.  Video  technicians  and  camera  operators  work  closely  with  trainers  who  are  out 
with  the  training  units,  to  mark  and  save  important  video  selections  to  be  used  in 
PowerPoint  presentations  in  the  AAR.  The  total  video  processing  time  required,  before 
the  data  can  be  used  for  the  AAR,  ranges  from  20-50  minutes  after  each  run,  a  significant 
delay  which  impacts  the  number  of  training  runs  that  can  occur  in  a  day  (Kobus,  2012). 

The  Contour  Camera  method  is  a  different  approach  to  capturing  video. 
The  trainer  employs  a  small  handheld  camera  and  selects  only  those  relevant  video 
streams  that  will  provide  the  maximum  value  to  the  training  audience.  The  video  streams 
are  then  directly  downloaded  from  the  memory  of  the  Contour  Camera  to  the  classroom 
computer  used  for  the  AAR.  Compared  to  the  TVCS  and  Video  Flashlight  system 
processing  time,  the  required  time  to  completely  process  the  selected  video  streams  is 
approximately  five  minutes.  Due  to  the  operational  requirements  of  the  training  units  and 
the  advantage  presented  by  the  Contour  Camera’s  processing  time,  the  Contour  Camera 
instrumented  AAR  method  is  now  the  preferred  method  of  instrumented  AARs.  The 
training  unit  uses  the  TVCS/Video  Flashlight  instrumented  PowerPoint  presentations  for 
archive  and  complementary  training  purposes. 


12 


D.  CURRENT  CAPABILITY  GAP  IN  ASSESSMENTS  IN  THE  DOD 

The  USMC  and  the  Department  of  the  Navy  have  recognized  specific  gaps  in  the 
manner  in  which  we  evaluate  and  train  our  Marines  and  Sailors,  particularly  in  live, 
virtual  and  constructive  (LVC)  environments.  Both  the  Marine  Corps  and  the  Navy  face 
difficulties  in  conducting  objective  evaluations  of  the  effectiveness  of  training  systems 
and  training  environments  in  terms  of  achieving  training  objectives.  This  ability  to 
objectively  evaluate  Marines  and  detennine  their  level  of  mastery  of  the  subject  matter 
has  been  identified  as  a  gap  in  current  performance  assessment  methods  that  include  an 
AAR  or  in  the  assessment  process  in  general  (Kobus,  2012;  ONR,  2011). 

Including  behaviorally-anchored  rating  scales  to  objectively  measure  if,  or  how 
well,  training  objectives  were  met  is  one  method  to  overcome  this  gap.  These  scales  can 
be  used  to  measure  the  performance  of  the  training  unit  and  the  results  can  either  be 
provided  during  the  AAR  or  reviewed  by  the  unit  leader  independent  of  the  AAR 
process.  This  discretion  afforded  to  the  training  unit  and  its  operational  tempo  provides 
flexibility  within  the  unit  leader’s  training  timeline.  However,  the  more  effective  method 
of  quickly  assessing  the  unit’s  level  of  mastery  of  training  objectives  would  involve 
incorporating  this  feedback  as  part  of  the  AAR.  Development  of  valid  and  reliable 
measures  and  metrics  will  facilitate  the  assessment  of  individual  and  collective 
performance  and  overall  training  effectiveness.  “These  measures  would  enhance  and 
validate  current  subjective  measures”  (Kobus,  2012,  p.  28). 

E.  MOTIVATION  AND  PURPOSE  OF  RESEARCH 

Research  in  the  domain  of  performance  assessment  and  its  impact  on  Marines  was 
motivated  by  a  strong  desire  to  have  a  positive  effect  on  the  Marine  Corps.  Aligning  this 
interest  in  perfonnance  assessment  with  a  documented  Marine  Corps  gap,  adheres  to  the 
CMC’s  guidance  that  Naval  Postgraduate  School  (NPS)  Marines  conduct  their  thesis 
research  in  areas  that  “align  with  the  USMC  needs”  (CMC,  2010A,  p.  14).  The  objective 
of  this  research  was  to  provide  military  trainers  with  behaviorally-anchored  measures  that 
can  be  used  to  provide  specific  feedback  to  trainees.  Research  conducted  for  this  thesis 
has  contributed  to  developing  tools  and  methods  to  quantifiably  assess  the  small-unit 
leader’s  decision-making  abilities. 
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F. 


BENEFITS  OF  RESEARCH 


Live  and/or  virtual-training  environments  such  as  the  IIT  provide  a  novel 
approach  to  train  and  evaluate  Marines  and  Sailors  in  a  dynamic,  immersive  environment 
approximating  realistic  combat  environments  (Smith  et  al.  2010).  The  results  of  this 
research  provide  the  trainer  with  measures  that  can  be  used  to  quantify  decision-making 
performance.  Current  budgetary  conditions  experienced  by  military  and  civilian  leaders, 
necessitate  an  effort  to  maximize  efficiencies  in  existing  programs  and  organizations 
(Bilmes,  2013).  Inculcating  the  lessons  learned  from  operational  training  exercises  into 
subsequent  training  evolutions  will  afford  greater  skill  retention  in  the  individual  and 
collective  warfighting  skill  sets  and  use  military  budgets  in  a  more  efficient  manner 
(Boese,  2013). 
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III.  LITERATURE  REVIEW 


A.  COGNITIVE  AND  CONSTRUCTIVIST  PSYCHOLOGY 

In  the  online  article  “More  Complex  Than  a  Galaxy”  -  New  Insights  Into  the 
Enormous  Biochemical  Complexity  of  the  Human  Brain,  Sir  Roger  Penrose  was 
referenced  comparing  the  complexity  of  the  human  mind  to  that  of  the  galaxy.  He  said, 
“consider  the  human  brain[,]  if  you  look  at  the  entire  physical  cosmos,  our  brains  are  a 
tiny,  tiny  part  of  it.  But  they’re  the  most  perfectly  organized  part.  Compared  to  the 
complexity  of  a  brain,  a  galaxy  is  just  an  inert  lump”  (as  cited  in  www.dailygalaxy.com, 
2012).  To  better  understand  the  human  mind,  modern  technology  and  science  have 
deconstructed  the  brain’s  organic  matter  into  its  biochemical  and  genetic  underpinnings 
(Hedden  &  Gabrieli,  2004).  While  these  endeavors  have  advanced  our  contemporary 
understanding  of  the  human  mind  and  its  capabilities,  there  is  a  plethora  of  outstanding 
questions  regarding  its  cognitive  processes  (McGinn,  1989).  Psychology  was  developed 
as  a  means  to  comprehend  these  cognitive  processes,  the  mind’s  relationship  to  human 
behavior,  and  its  connection  with  the  external  environment  and  society  (Anderson,  2009). 

Our  modem  understanding  of  the  human  mind  and  cognition  came  from  centuries 
of  philosophical  literature,  theoretical  and  empirical  research,  and  advancements  in 
technology  and  neuroscience  (Finger,  2001).  Ulric  Neisser  formalized  the  domain  of 
cognitive  psychology  with  his  1967  book,  “Cognitive  Psychology”  (Neisser,  1967). 
Neisser’s  work  was  fundamentally  influenced  by  Noam  Chomsky’s  1959  critique  of  B.F. 
Skinner’s  “Verbal  Behavior”  (world’s  leading  behaviorist  theorist’s  book  on  language). 
In  his  assessment  of  Skinner’s  work,  Chomsky  provided  an  “effective  refutation  of  the 
behavioristic  approach  to  language”  (Neisser,  1967,  p.  246).  Neisser  presented 
cognitivism  as  a  more  suitable  and  holistic  approach  to  knowledge  elicitation.  He 
emphasized  that  a  strict  adherence  to  behavioral  response  or  association  to  external 
stimulus  was  insufficient  to  explain  the  mind’s  mental  processes  (Neisser,  1976). 

Nessier  defined  cognition  as  the  “sensation,  perception,  imagery,  retention,  recall, 
problem-solving,  and  thinking”  aspects  of  mental  processes  (Neisser,  1967,  p.  4).  The 
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cognitivist  “emphasize[s]  individual  thinkers  and  their  isolated  minds”  in  the  absence  of, 
“the  social  nature  of  cognition  and  meaning”  (Jonassen  &  Land,  2000,  p.  26).  A  central 
tenet  of  cognition  is  metacognition  or  “Knowing  about  Knowing”  (“Metacognition,”  n.  d.). 

Metacognition  refers  to  one’s  knowledge  concerning  one’s  own  cognitive 
processes  or  anything  related  to  them,  e.g.,  the  learning-relevant  properties 
of  information  or  data.  For  example,  I  am  engaging  in  metacognition  if  I 
notice  that  I  am  having  more  trouble  learning  A  than  B;  if  it  strikes  me 
that  I  should  double  check  C  before  accepting  it  as  fact.  (Nisbet  & 
Shucksmith,  1984,  p.  6) 

Metacognition  is  evident  when  learners  engage  in  training  involving  ambiguous 
scenarios  with  insufficient  information.  Effective  outcomes  are  achieved  when  learners 
comprehend  what  is  necessary  to  overcome  the  obstacle(s)  to  achieving  their  objectives 
and  adapt  their  decision-making  strategies  accordingly  (Morrison  &  Fletcher,  2002; 
Fortunato,  1991).  Training  conducted  at  the  infantry  immersion  trainer  (IIT)  Camp 
Pendleton,  CA,  employs  this  cognitivist  approach  (Kobus  &  Viklund,  2012). 

Cognitivist  theory  evolves  into  the  Constructivist  paradigm  by  incorporating  the 
internal  symbolic  meaning  of  external  social  and  environmental  relationships.  The 
constructivist  rejects  the  notion  that  the  learner  perceives  reality  from  the  external 
environment;  rather  reality  is  internally  constructed  based  upon  the  learner’s  perceptual 
experiences  (Cooper,  1993).  “Understanding  is  an  individual  construction,  we  cannot 
share  understandings  but  rather  we  can  test  the  degree  to  which  our  individual 
understandings  are  compatible”  (Savery  &  Duffy,  1995,  p.  1).  During  after  action 
reviews  (AARs),  Marines  verbally  recollect  the  events  of  the  scenario  and  present  their 
interpretations  of  reality.  Providing  multiple  facets  of  understanding  and  perspective 
enhances  the  shared  understanding  of  events.  This  dissemination  of  various  perspectives 
fosters  development  of  common  situational  awareness  among  the  individual  Marines  and 
affords  them  the  opportunity  to  iteratively  correct  their  misinterpretations  of  events 
presented  in  the  training  scenario  (Smith  et  ah,  2010). 


16 


B.  ADDITIONAL  IMPLICATIONS  OF  COGNITIVE  AND 

CONSTRUCTIVIST  PSYCHOLOGY 

Cognitive  and  constructivist  science  naturally  apply  to  the  fields  of  educational, 
abnormal,  social,  developmental,  and  personality  psychology  (“Cognitive  Psychology,” 
n.  d.).  These  specializations  within  psychology  provide  a  broader  applied  discipline 
involving  “the  scientific  study  of  the  behavior  of  individuals  and  their  mental  processes” 
(“Psychology,”  n.  d.).  Due  to  the  limited  scope  of  this  thesis,  the  focus  of  this  chapter 
will  be  on  cognitive  and  constructivist  influences  on  educational  and  social  psychology. 

1.  Educational  Psychology 

In  instructional  enviromnents,  learners  and  instructors  inherently  possess  a  variety 
of  biases,  aptitudes,  experiences,  and  sociocultural  backgrounds.  Instructional  design  of 
the  learning  environment  and  the  relevant  cognitive  learning  theory  must  be  tailored 
according  to  these  individual  differences  (Bereiter,  1990;  Brook  &  Ross,  2002).  This 
dynamic,  current  learning  theory  is  divided  into  three  levels  of  learning  strategies.  At  the 
lowest  level  are  learning  theories  based  on  genetics  and  neuroscience.  Theories  based  on 
a  psychological  context  constitute  the  medium  level  and  philosophical  and  secular  based 
contexts  comprise  the  most  complex  and  highest  level  of  learning  theories  (Brook  & 
Ross,  2002).  Categorization  in  this  manner  aids  facilitation  of  an  optimal  learning 
strategy,  however  the  strategy  does  not  include  the  additional  and  necessary  external 
relationships  included  in  constructivism.  The  intention  of  this  discussion  is  not  to 
highlight  the  differences  or  refute  the  merits  of  cognitivism;  rather  the  goal  is  to 
emphasize  their  complementary  natures  (Bereiter,  1990).  Both  the  cognitivist  and 
constructivist  approaches  present  an  end  state  where  the  learner  perceives,  analyzes, 
comprehends  and  then  decides  on  suitable  courses  of  actions  (COAs)  (Gelder,  2005).  The 
optimal  outcome  of  either  approach  is  an  effective  byproduct  that  is  efficiently  employed 
by  the  instructor. 

2.  Social  Psychology 

Humans  are  intrinsically  social  creatures  and  are  beholden  to  societal  norms  and 
traditions.  Comprehending  verbal  and  non-verbal  interactions  (such  as,  gesture,  body 
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language,  posture,  tone  of  voice  or  facial  expressions)  is  essential  to  fonn  social  identity 
and  relationships  with  individuals  and  groups  (Howard,  2000).  Social  psychology  is  “the 
branch  of  psychology  that  studies  the  effect  of  social  variables  on  individual  behavior, 
attitudes,  perceptions,  and  motives  [,  and]  also  studies  group  and  intergroup  phenomena” 
(“Social  Psychology,”  n.  d.).  Social  psychologists  attempt  to  understand  individual  and 
group  social  interactions  through  cognitivist  and  constructivist  paradigms  (Reich,  2003; 
Reicher,  2004). 

The  concept  of  identity  is  critical  to  social  standing  and  purpose  within  the  group. 
Organized  individuals  have  a  cognitive  need  to  efficiently  categorize  individuals,  groups, 
events,  and  objects  in  order  to  reduce  their  cognitive  workload  (Howard,  2000).  In 
highly  organized  groups  such  as  Marine  squads  and  platoons,  comprehending  each 
Marine’s  role  and  function  is  essential  to  mission  accomplishment.  These  classifications 
aggregate  complex  and  fluid  relationships  into  discrete  elements  which  aid  in  recollection 
and  utility.  Marine  units  operating  in  highly  complex  and/or  ambiguous  situations  rely 
on  this  identity  simplification  as  a  means  to  mutually  support  common  objectives 
(Zaccaro,  Rittman,  &  Marks,  2002). 

Cognitive  readiness  [refers  to]  the  mental  preparation  (including  skills, 
knowledge,  abilities,  motivations,  and  personal  dispositions)  an  individual 
needs  to  establish  and  sustain  competent  performance  in  the  complex  and 
unpredictable  environment  of  modern  military  operations.  (Morrison  & 
Fletcher,  2002,  p.  ES-1) 

Within  the  cognitive  readiness  domain,  there  are  generally  accepted 
characteristics  or  heuristics  that  support  effective  decision-making.  The  following 
paragraphs  describe  the  United  States  Marine  Corps  (USMC)  cognitive  readiness 
initiatives  related  to  decision-making  at  the  small  unit  level. 

3.  United  States  Marine  Corps  Cognitive  Readiness  Initiatives  and 
Research 

In  2008,  Commandant  of  the  Marine  Corps  (CMC),  Gen.  James  T.  Conway 
(USMC),  promulgated  his  “vision  of  the  future  Corps  and  a  plan  for  creating  the  Marine 
Corps  of  2025”  (Conway,  2008,  p.  3).  This  document  informed  Marines,  combatant 
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commanders,  and  civilian  leadership  of  the  future  direction  the  Marine  Corps  would 
provide  to  national  defense.  Additionally,  it  prioritized  strategic  objectives  in  order  to 
“respond  rapidly  and  decisively  to  crises  anywhere  in  the  world”  (Conway,  2008,  p.  14). 
One  of  these  objectives  established  the  need  to  train,  educate,  and  prepare  Marines  for 
expeditionary  operations  in  challenging  environments.  This  Marine  Corps  document 
maintains  that  additional  resources  provided  to  noncommissioned  (NCO)  and  junior 
officer  grades  will  assist  acclimatization  to  decentralized  combat  operations.  This 
professional  military  education  will  condition  ethical  and  sound  decision-making  skills  in 
small-unit  leaders. 

The  Marine  Corps  Vision  &  Strategy  2025  compendium  is  a  planning  document 
that  will  be  promulgated  in  the  following  years  in  order  to  bring  the  CMC’s  intent  from 
concept  to  reality.  The  implementation  plan  organized  this  strategy  into  three  future 
years  defense  plan  groups:  “near-term:  present  [2008]  -  2011;  mid-term:  2012  -  2016; 
and  far-term:  2017  and  beyond”  (Conway,  2008,  p.  1).  The  document’s  first  task  directed 
the  development  of  diverse  approaches  to  NCO  and  junior  officer  intuitive  decision¬ 
making.  Consistent  with  Klein’s  naturalistic  decision-making  paradigm,  small  unit 
leaders  will  be  trained  to  effectively  recognize  cues  and  patterns  from  the  cultural  and 
environmental  terrain  in  order  to  acquire  expertise.  Current  and  emergent  decision¬ 
making  initiatives  were  codified  in  this  implementation  guidance  (Conway,  2008). 

In  January  2011,  the  small  unit  decision-making  (SUDM)  initiative,  under  the 
purview  of  United  States  Marine  Corps  Training  and  Education  Command  (TECOM), 
conducted  a  workshop  that  focused  on  identifying  tools,  techniques,  and  procedures  to 
support  the  SUDM  goals  (TECOM,  2011).  The  goal  of  the  SUDM  initiative  is  to 
accelerate  cognitive  readiness  in  novice  small-unit  leaders  and  inculcate  its  tenets  in  their 
subordinates  (Schatz,  et  ah,  2012).  The  SUDM  workshop  brought  together  leading 
cognitive  researchers,  scientists,  and  academics  along  with  Marine  officers  who  work  in 
PME  development  and  instruction.  Workshop  participants  collectively  agreed  that  the 
SUDM  initiative  and  its  goals  would  only  be  achieved  through  sustained  planning  and 
commitment.  Additionally,  they  identified  five  competencies  of  cognitive  readiness  that 
support  effective  decision-making. 
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As  listed  and  defined  in  Table  1,  these  cognitive  readiness  competencies 
(indicated  by  the  shaded  cells)  are  sensemaking,  adaptability,  attention  control, 
metacognition,  and  problem  solving  (Schatz,  et  ah,  2012;  TECOM,  2011).  Table  1  also 
includes  definitions  for  ten  cognitive  and  relational  skills  (CARS)  (indicated  by  the  non- 
shaded  cells)  which  represent  a  finer  resolution  of  the  five  cognitive  competencies  and 
“indicate  decision-making  proficiency”  (Ross  et  ah,  2012,  p.  1-2;  TECOM,  2011).  Each 
of  the  cognitive  competencies  and  CARS  represent  different  dimensions  of  cognitive 
readiness.  Each  attribute  addresses  a  particular  gap  of  decision-making  during  military 
operations. 
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Table  1.  TECOM  Small  Unit  Decision-Making  Cognitive  Competencies  and  CARS 

(From  Ross  et  al.,  2012,  p.  A2-A14). 

Sensemaking:  The  cognitive  process,  driven  by  a  specific  goal,  of  filtering  information 
for  relevancy  and  using  it  to  construct  and  continually  assess  an  explanation  of  the  broad 
or  specific  situation,  often  in  the  form  of  a  story,  in  order  to  understand  how  and  why  the 
situation  evolved  and  anticipate  what  might  happen  next. _ 

Perspective  Taking:  Visualizing  the  situation  from  another’s  viewpoint  and 
assessing  his  or  her  motivations  and  objectives,  to  predict  his  or  her  future  actions  and 
proactively  position  for  or  take  advantageous  action. _ 

Analytical  Reasoning:  Critically  and  deliberately  examining,  assessing,  and 
critiquing  one  or  more  alternatives  or  assumptions  in  the  context  of  specified  goals 
(e.g.,  the  mission)  and  against  a  set  of  evaluative  criteria  (e.g.,  intent,  timing, 
resources,  or  ROE). _ 

Anomaly  Detection:  Critically  and  deliberately  examining,  assessing,  and  critiquing 
one  or  more  alternatives  or  assumptions  in  the  context  of  specified  goals  (e.g.,  the 
mission)  and  against  a  set  of  evaluative  criteria  (e.g.,  intent,  timing,  resources,  or 
ROE). _ 

Situational  Assessment:  Analytically  or  intuitively  identifying  and  collecting 
information  from  multiple  available  sources,  including  one’s  own  knowledge,  to 
analyze  relevant  factors  of  METT-TC  (mission,  enemy,  terrain  and  weather,  troops 
and  support  available,  time  available,  civil  considerations)  and  construct  an 
understanding  of  the  situation  to  support  a  specific  task  or  goal. _ 

Adaptability:  Fluidly  modifying  or  changing  one’s  planned  actions  when  the  situation 
has  changed  from  what  was  expected,  or  when  the  typical  approach  or  plan  is  rendered 
less  effective  than  necessary. _ 

Situational  Assessment:  Analytically  or  intuitively  identifying  and  collecting 
information  from  multiple  available  sources,  including  one’s  own  knowledge,  to 
analyze  relevant  factors  of  METT-TC  and  construct  an  understanding  of  the  situation 
to  support  a  specific  task  or  goal. _ 

Cognitive  Flexibility:  Applying  knowledge  and  principles  of  tactics  and  leadership 
differentially  based  upon  the  unique  demands  of  the  situation.  Applying  knowledge 
learned  in  one  context  to  multiple  relevant  contexts. _ 

Ambiguity  Tolerance:  The  ability  to  calmly  withstand  and  operate  within  uncertain 
environments  by  delaying  drawing  a  conclusion  or  making  a  decision,  or  by  making 

assessments  and  decisions  in  the  face  of  uncertainty. _ 

Continued  on  next  page 
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_ Continued  from  previous  page _ 

Attentional  Control:  Activities  related  to  maintaining  a  focus  on  mission  completion 
despite  distracters  including  stress,  boredom,  fatigue,  and  emotion. _ 

Resilience:  Overcoming  the  stress,  fatigue,  emotion,  or  pain  associated  with  a  current 
or  past  event  or  situation  in  order  to  maintain  or  return  to  effectiveness  as  a  leader  and 
decision  maker. _ 

Self-Regulation:  Monitoring,  assessing,  and  adjusting  one’s  own  behavior  and  its 
effects  in  order  to  impact  the  situation  in  a  way  that  supports  mission,  unit,  or  training 
goals. _ 

Metacognition:  Activities  related  to  considering  one’s  own  thought  processes,  including 
assessments  of  strengths  and  limitations  or  developmental  needs,  in  support  of 
performing  or  learning  the  job. _ 

Self-Awareness:  Conscious  knowledge  of  one’s  own  character,  motives,  knowledge 
base,  and  skill  set  in  order  to  request  information  or  assistance  when  the  requirements 
of  the  situation  call  for  capabilities  beyond  one’s  current  abilities. _ 

Problem  Solving:  Identification,  definition,  examination,  prioritization,  and  resolution  of 
situations  that  impede  task  or  mission  accomplishment. _ 

Analytical  Reasoning:  Critically  and  deliberating  examining,  assessing,  and 
critiquing  one  or  more  alternatives  or  assumptions  in  the  context  of  specified  goals 
(e.g.,  the  mission)  and  against  a  set  of  evaluative  criteria  (e.g.,  intent,  timing, 
resources,  or  ROE). _ 
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Proficiency  in  each  of  the  cognitive  competencies  and  CARS  will  enhance  and 
support  the  decision-making  performance  of  the  individual  unit  leader  (Ross  et  ah,  2012). 
Performance  in  this  context  does  not  exclusively  apply  to  the  training  and/or  garrison 
environment.  Increasing  the  unit  leader’s  performance  in  the  training  environment  must 
have  a  positive  correlation  with  expeditionary  operations  (Morrison  &  Fletcher,  2002). 
Attaining  a  level  of  competence  in  cognitive  readiness  should  have  an  equal  impact  on 
operational  effectiveness  (Walsh  &  Shingledecker,  2006).  Cognitive  readiness  is  viewed 
as  an  interventional  process  which  is  assessed  and  measured  against  its  product  (Morrison 
&  Fletcher,  2002). 

C.  DECISION-MAKING  THEORIES 

Cognitive  readiness  is  a  concept  that  provides  a  set  of  attributes  that  can  be  used 
to  prepare  Marines  to  function  effectively  given  the  external  influences  from  the 
environment.  The  decision  to  act  or  not  act  in  operational  environments  is  pivotal  given 
the  life  and  death  nature  of  combat.  Decision-making  theories  promote  an  understanding 
of  individual  and  organizational  decision-making  processes  with  respect  to  their 
outcomes.  Decision-making  theories  discussed  in  the  remainder  of  this  chapter  will  focus 
on  intuitive  strategies. 

In  decision-making  situations  that  are  extremely  complex,  ambiguous,  or  time- 
constrained,  a  more  streamlined  decision-making  process  is  typically  employed.  The 
rational  decision-making  approach  is  not  applicable  for  highly  uncertain  or  time 
constrained  decision-making  situations.  In  fast-paced,  uncertain  decision-making 
situations  a  “satisficing”  decision  strategy  is  employed  (Simon,  1957,  p.  204-205). 
Satisficing  reduces  the  learner’s  cognitive  workload  by  shedding  non-essential  tasks  from 
consideration  and  allows  the  learner  to  focus  on  the  first  sufficient  COA  rather  than  the 
optimal.  The  learner’s  pattern  recognition  capability  is  an  important  discriminator 
between  the  decision-making  abilities  of  a  novice  and  that  of  an  expert.  The  following 
discussion  of  decision-making  theories  is  not  meant  to  be  all-inclusive;  rather  theories 
discussed  in  the  following  sections  are  considered  to  be  most  applicable  to  this  thesis 
research. 
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1.  Prospect  Theory 

Through  much  of  the  20th  century,  decision-making  theory  was  dominated  by 
expected  utility  theory  (EUT)  (Kahneman  &  Tversky,  1979).  EUT,  postulated  by 
Bernoulli  in  1738,  and  later  refined  in  1947  by  Von  Neumann-Morgenstern,  explained 
individuals  in  terms  of  the  “rational  decision  maker”  (“Expected  Utility  Hypothesis,”  n. 
d.;  “Von  Neumann-Morgenstern  utility  theorem,”  n.  d.).  The  rational  or  normative 
model  of  decision-making  described  how  individuals  probabilistically  decided  on  the 
most  preferred  action  given  all  the  information.  While  concise  and  explanatory,  the 
theory’s  rationale  did  not  provide  an  adequate  explanation  for  decision-making  situations 
that  involve  uncertainty  and  preference.  In  1979,  Kahnman  and  Tversky  published  a 
critique  of  EUT  through  their  prospect  theory  of  decision-making.  This  theory’s  critique 
of  EUT  was  based  on  empirical  examples  of  economic  behavior  related  to  gambling  and 
insurance  (Kahneman  &  Tversky,  1979).  Prospect  theory  fundamentally  contributed  to 
decision-making  and  “demonstrate [d]  that  human  judgment  arises  from  qualitatively 
different  processes  than  suggested  by  nonnative  theories,  and  argue[d]  that  heuristics 
would  provide  a  better  starting  point  for  the  development  of  psychological  theory” 
(Phillips,  Klein,  &  Sieck,  2004,  p.  297-298). 

2.  Recognition-primed  Decision  Model 

In  1989,  the  recognition-primed  decision  (RPD)  model  was  introduced  by  Klein, 
Calderwood,  and  Clinton-Cirocco.  The  model  explains  how  people  make  decisions 
expeditiously  without  comparing  multiple  options.  It  describes  a  naturalistic  method  of 
recognizing  the  “plausible  [and  suitable  choice]  as  the  first  one  to  consider”  (Ross,  Klein, 
Thunhohn,  Schmitt,  &  Baxter,  2004,  p.  6).  Klein  and  associates  developed  the  RPD 
model  after  observing  experienced  firefighters  and  interviewing  them  regarding  their 
decision-making  processes.  They  discovered  that  these  firefighters  never  made  explicit 
decisions;  instead  they  instinctually  acted  upon  the  ground  truths  of  the  situation  (Phillips 
et  ah,  2004).  That  is  to  say,  firefighters  were  able  to  recognize  key  aspects  of  the 
situation,  based  on  prior  experience  with  similar  situations,  and  were  able  to  come  up 
with  a  workable  solution.  Through  experiential  recognition,  firefighters  recalled  previous 
similar  events,  their  associated  solutions,  and  implicitly  decided  upon  the  most  workable 
COA. 
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As  depicted  in  Figure  2,  two  critical  components  of  the  RPD  model  are  situation 
assessment  and  mental  simulation.  Both  processes  provide  the  means  to  develop  a 
particular  COA  and  assess  its  validity.  The  pedagogical  intent  of  RPD  is  to  condition 
individuals  to  effectively  synthesize  meaningful  connections  from  previous  experiences, 
and  provide  timely  and  efficient  correlations  to  the  current  solution  space.  When  the 
decision  maker  uses  a  RPD  strategy  he/she  does  not  systematically  process  all 
combinatorial  paths  or  probabilities  in  order  to  rationally  decide;  rather  using  a  RPD 
decision  strategy  entails  selecting  a  COA  that  suffices  given  the  limitations  and 
constraints  of  the  real-world  (Klein  &  Klinger,  1991). 


A.  Simple  Match 


d 


Experience  the  Situation^ 


B.  Developing  a  Course 
of  Action 


C.  Complex  RPD  Strategy 


Figure  2.  RPD  Model  (From  Klein  and  Klinger,  1991,  p.  18). 


3.  Observe-Orient-Decide-Act  Loop 

In  1977,  the  observe-orient-decide-act  (OODA)  loop,  or  Boyd’s  theory,  was  first 
presented  in  Col.  John  R.  Boyd’s  (USAF),  “Patterns  of  Conflict”  (Osinga,  2005,  p.  2; 
Boyd,  1986).  This  theory  represented  the  culmination  of  Boyd’s  work  regarding  rapid 
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decision-making  in  uncertain  and  competitive  environments.  Boyd’s  theory  describes  the 
individual’s  decision-making  cycle  as  the  means  to  adapt,  maneuver,  and  survive  in  a 
belligerent  and  resource-constrained  environment  (Osinga,  2005).  Boyd  describes  this 
competition  as  a  contest  of  wills,  where  participants  prove  to  be  successful  or 
unsuccessful  relative  to  the  timeliness  and  acuity  of  their  decisions  (Boyd,  1976). 

As  depicted  in  Figure  3,  during  the  observation  stage,  perceived  sensory 
information  from  the  external  environment  is  processed  and  provides  the  necessary  input 
for  the  orientation  stage.  The  individual’s  experiences,  biases,  and  background  (physical, 
moral,  and  cultural)  provide  the  necessary  contextual  reference  and  meaning  for  that 
information.  The  synthesized  product  is  then  forwarded  to  the  decide  stage,  where 
judgment  and  deliberation  are  used  to  either  reject  or  accept  the  COA.  Once  the  COA  is 
implemented,  subsequent  feedback  loops  ensure  the  actions  are  aligned  with  the  decision 
made  (Ullman,  2007). 


Observe  Orient  Decide  Act 


Figure  3.  Boyd’s  OODA  Loop  (From  Ullman,  2007,  p.  2). 


4.  Cognitive  Observe-Orient-Decide-Act  Model 

The  cognitive  observe-orient-decide-act  (C-OODA)  model  is  a  natural  extension 
of  the  OODA  Loop  model.  As  shown  in  Figure  3,  Boyd’s  model  was  a  “high-level 
abstraction  of  the  decision  cycle”  (Hannigan  et  ah,  2012,  p.  6).  The  critical  limitation  of 
the  OODA  Loop  was  its  presentation  of  the  cognitive  decision-making  process.  The 
original  model  was  deficient  in  ascertaining  the  necessary  cognitive  requirements  used  to 
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identify  the  “dynamic  properties  of  the  decision-making  task”  (Breton  &  Rousseau,  2007, 
p.  243).  Additional  limitations  regarding  solitary  input  channels  and  granularity  of 
decision-making  expertise  required  further  research  (Hannigan  et  ah,  2012). 

The  C-OODA  model  resolves  the  limitations  noted  above  by  integrating  three 
applicable  cognitive  theories  into  the  observation,  orientation,  and  decision  phases  of  the 
OODA  model:  (the  act  phase  is  purposely  excluded  due  to  the  additional  resource  and 
skill  requirements)  (Hannigan  et  ah,  2012).  The  C-OODA  loop  integrates  Treisman’s 
feature  integration  model,  Endsley’s  model  of  situation  awareness,  and  the  RPD  model  as 
the  observation,  orientation,  and  decision  phases  respectively  (Endsley,  1995;  Hannigan 
et  ah,  2012;  Klein,  1988;  Treisman  &  Gelade,  1980).  As  depicted  in  Figure  4,  the  C- 
OODA  loop  incorporates  a  higher-  and  lower-level  of  cognitive  processing.  The  lower- 
level  processing  applies  uniformly  to  all  skill  levels  (novice  to  expert  decision  makers), 
while  the  higher-level  processing  is  apparent  when  proficiency  and  expertise  mature 
(Hannigan  et  ah,  2012). 


Figure  4.  C-  OODA  Loop  (From  Hannigan  et  ah,  2012,  p.  7). 

As  depicted  in  Figure  4,  lower-level  mental  processing  of  perceived  sensory 
stimuli  occurs  through  feature  matching  and  experiential  association.  Correlated  objects 
(information)  of  interest  are  filtered  out  and  forwarded  to  the  next  phase  of  the  C-OODA 
loop  where  further  processing  of  the  objects  will  occur  (Treisman  &  Gelade,  1980). 
Once  the  selected  information  is  forwarded  to  the  orientation  phase,  lower-level 
processing  and  comprehension  of  the  information’s  importance,  priority,  and  relevance 
occur  (Endsley,  1995).  Once  the  information  is  comprehended,  and  then  referenced  with 
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respect  to  the  individual’s  norms,  goals,  and  biases,  relationships  are  established  to 
facilitate  projection  of  the  individual’s  knowledge.  Finally,  information  is  forwarded  to 
the  decide  phase,  where  the  decision  maker  becomes  cognizant  of  the  information’s 
relationship  with  respect  to  its  importance  or  familiarity  (Klein  &  Klinger,  1991).  Once 
that  understanding  is  achieved,  individuals  must  evaluate  the  potential  COA  and  the  end 
state  it  will  produce.  Once  this  COA’s  evaluation  process  is  complete  and  the  COA’s 
decision  criterion  is  met,  the  COA  is  forwarded  to  the  act  phase. 

5.  Maneuver  Squad  Leader  Mastery  Model 

The  Maneuver  Squad  Leader  Mastery  Model  (MSLMM)  is  a  “five-stage 
descriptive  model”  that  is  “designed  to  (1)  provide  insights  into  how  individuals 
progressively  develop  into  high  performing  maneuver  squad  leaders,  and  (2)  provide 
implications  for  what  should  be  assessed  and  how  during  development”  (Ross  et  al., 
2012,  p.  2).  The  model  was  developed  under  TECOM  sponsorship  for  the  SUDM 
initiative.  The  goal  was  to  explain  the  process  of  optimizing  attributes  of  expertise  and 
decision-making  competence  as  a  “maneuver  squad  leader”  (Ross  et  al.,  2012,  p.  1).  The 
model  can  be  used  to  assess  the  maneuver  squad  leader  (MSL)  across  nine  performance 
areas  via  five  cognitive  competencies  and  ten  CARS  (see  Table  1.). 

Behaviorally-anchored  rating  scales  (BARS)  that  classify  observable  behavior 
with  quantitative  scales  are  incorporated  in  the  MSLMM  (Ross  et  al.,  2012).  The 
MSLMM  addresses  the  need  to  describe  the  decision-making  process  with  respect  to  the 
maneuver  squad  leader  through  incremental  stages  (novice  to  expert).  BARS  used  in  the 
MSLMM  were  generated  through  the  situational  judgment  test  and  decision  requirements 
interview  instruments.  The  MSLMM  requires  future  pilot  studies  to  refine  the 
assessment  battery  (incorporating  BARS)  beyond  the  currently  restricted  academic 
domain  (Ross  et  al.,  2012). 

D.  PERFORMANCE  MEASURES  DEVELOPMENT 

The  goal  of  this  thesis  is  to  develop  performance  measures  that  provide  the 

evaluator  a  more  objective  means  than  currently  used  to  assess  the  decision-making  of 

Marine  squad  leaders  in  training  environments.  Assessing  individual  and  unit 
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performance  is  “an  essential  part  of  team  training  and  training  research”  (Shanahan,  Best, 
Finch,  &  Sutton,  2007,  p.  1).  Factors  that  influence  individual  and  unit  performance  and 
operational  effectiveness  can  be  qualitative  or  quantitative,  singular  or  complex  in  nature 
(Schwab,  Heneman,  &  DeCotiis,  1975).  Examples  of  these  factors  that  influence 
individual  and  unit  performance  are  explained  in  the  cognitive  readiness  section  of  this 
chapter.  While  the  list  of  attributes  in  Table  1  is  not  all-inclusive,  it  does  provide 
examples  of  cognitive  and  behavioral  attributes  used  in  the  assessment  and  analysis  of 
decision-making. 

This  thesis  views  performance  measures  through  the  prism  of  individuals 
operating  in  organized  units.  Individuals  effectively  operating  in  organized  units  should 
exhibit  competence  related  to  “(1)  infonnation  exchange,  (2)  communication,  (3) 
supporting  behavior,  and  (4)  initiative  and  leadership”  (Shanahan  et  ah,  2007,  p.  5). 
These  behaviors  and  the  underlying  cognitive  processes  that  support  these  behaviors  must 
be  assessed  within  the  context  of  the  task,  objective,  and  the  environment  where  the  task 
was  performed  (Phillips,  Shafer,  Ross,  Cox,  &  Shadrick,  2006A).  Diagnostically,  the 
behavior  must  be  distinct  enough  to  be  discriminable,  while  selective  enough  to  allow  for 
an  acceptable  amount  of  variability  from  the  target  audience  (novice  to  expert).  The 
behavioral  assessment  and  its  correlated  metric  must  provide  the  evaluator  the  means  to 
succinctly  and  efficiently  annotate,  audit,  and  catalog  adherence  to  or  deviations  from  the 
performance  standard.  Balancing  the  sensitivity  and  utility  of  the  measurement  tool 
requires  due  diligence  in  the  design  process,  comprehension  of  the  task  and  objective, 
and  concurrence  with  subject-matter  expertise  (Phillips,  Ross,  &  Shadrick,  2006B). 
Human  interaction  factors  must  be  considered  in  the  design  and  development  of  the 
measure  along  with  implementation  of  the  tool.  The  following  sections  on  cognitive  task 
analysis  (CTA)  and  BARS  describe  the  methodology  used  to  develop  performance 
measures  for  this  thesis  research.  The  subsequent  method  chapter  will  provide  greater 
detail  regarding  these  two  phases. 
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1.  Cognitive  Task  Analysis 

Cognitive  task  analysis  (CTA)  refers  to  a  group  of  methods  that  are  extensively 
used  in  cognitive  systems  engineering  and  naturalistic  decision-making  applications 
(Hutchins,  Pirolli,  &  Card,  2004).  Klein’s  (2001)  definition  of  CTA  is  “methods  for 
capturing  expertise  and  making  it  accessible  for  training  and  system  design”  (p.  173). 
Klein  delineated  the  following  five  steps:  (a)  identifying  sources  of  expertise,  (b) 
assaying  the  knowledge,  (c)  extracting  the  knowledge,  (d)  codifying  the  knowledge,  and 
(e)  applying  the  knowledge.  System  design  goals  supported  by  CTA  include  human- 
computer  interaction  design,  developing  training,  tests,  and  models  to  serve  as  a 
foundation  for  developing  an  expert  system,  and  analysis  of  a  team’s  activities  to  support 
allocation  of  responsibilities  to  individual  humans  and  cooperating  computer  systems. 
(An  extensive  treatment  of  CTA  methodology  appears  in  Crandall,  Klein,  &  Hoffman, 
2006.) 

Different  CTA  methods  are  used  for  different  goals  (Hoffman,  Shadbolt,  Burton, 
&  Klein,  1995.)  My  goal  for  conducting  a  CTA  was  to  obtain  operational  expertise  from 
SMEs  on  task  performance  of  complex  tasks  that  are  trained  in  the  IIT  and  use  this 
information  to  develop  behaviorally-anchored  performance  measures.  A  CTA  requiring 
resource-intense  analysis  is  characterized  by  complex,  ill-structured  tasks  that  are 
difficult  to  learn,  complex,  dynamic,  uncertain,  and  real-time  environments  as  well  as 
multitasking.  A  CTA  is  appropriate  when  the  task  requires  the  use  of  a  large  and  complex 
conceptual  knowledge  base,  the  use  of  complex  goal-action  structures  dependent  on  a 
variety  of  triggering  conditions,  or  complex  perceptual  learning  or  pattern  recognition 
skills.  The  tasks  performed  by  Marine  Corps  squads  involve  all  of  these  characteristics. 

The  CTA  that  was  planned  for  this  research  would  employ  one  of  the  two 
methods  traditionally  used  for  knowledge  elicitation  from  SMEs.  These  techniques  are 
the  critical  decision  method  (CDM)  (Hoffman,  Crandall,  Shadbolt,  1998)  and  the 
knowledge  audit  (KA)  (Militello  &  Hutton,  1998).  The  CDM  generally  asks  experts  to 
recall  a  critical  incident  and  asks  them  to  annotate  decision  nodes  through  a  constrained 
timeline  and  draw  out  possible  alternative  vignettes  through  persistent  inquiry.  KA  is  a 

“set  of  questions  or  probes  that  have  been  extracted  from  the  literature  of  expert-novice 
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differences”  (Craig  et  al.,  2012,  p.  1027).  The  KA  technique  reveals  noteworthy  items  in 
the  subject  matter,  such  as,  a  macro-level  perspective  of  the  events,  meaningful  signals 
from  the  environment,  and  how  an  expert  would  improvise  when  responding  to  presented 
vignettes.  The  KA  provides  expeditious  data  collection  and  implementation  advantages 
with  respect  to  CDM,  but  lacks  the  depth  and  detail  that  is  required  for  CTAs  not 
constrained  in  acute  time  and  resource  environments  (Fowlkes,  Salas,  Baker,  Cannon- 
Bowers,  &  Stout,  2000).  The  initial  plan  to  conduct  a  CTA  was  altered.  The  ONR 
sponsor  of  this  Marine  Corps  work  directed  our  attention  to  a  recent  CTA  report  on  IED- 
defeat  (Ross  et  al,  2012).  The  research  plan  was  modified  with  the  intent  to  make  use  of 
the  infonnation  included  in  the  CTA  on  IED  defeat 

2.  Behaviorally  Anchored  Rating  Scales 

Behavioral  measures  are  defined  as  “performance  dimensions  and  scale  values  in 
behavioral  terms”  (Schwab,  Heneman,  &  DeCotiis,  1975,  p.  550).  These  measures,  when 
compared  to  qualitative  trait  measures,  provide  the  evaluator  more  objective  measures  to 
assess  the  trainees’  perfonnance.  With  measures  that  are  less  subjective,  the  evaluator 
can  maintain  the  necessary  objectivity  during  performance  assessment.  Behaviorally- 
anchored  rating  scales  (BARS)  were  developed  as  a  means  to  assess  the  individual’s 
performance  in  “multidimensional,  behavior-specific  terms”  (Schwab  et  al.,  1975,  p. 
550).  While  the  BARS  concept  is  applicable  to  multiple  domains,  BARS  developed  for 
this  thesis  research  are  meant  for  “scenario-based,  training  sessions  requiring 
sensemaking  and/or  decision-making,  including  [...]  tactical  decision  game  or  decision¬ 
making  exercise  sessions”  (Phillips  et  al.,  2006A,  p.  1). 

In  developing  BARS,  there  is  a  uniform  methodology  be  used  for  their 
construction.  Subject  matter  experts  provide  infonnation  regarding  gradations  of 
expertise  (novice  to  expert)  and  efficacy  (ineffective  to  highly  effective)  of  outcome 
(Schwab  et  al.,  1975).  This  knowledge  elicitation  procedure  is  normally  conducted  using 
the  CTA  method.  Results  of  the  CTA  are  used  to  develop,  identify,  classify,  and  group 
behavioral  attributes  according  to  the  event.  Once  complete,  a  separate  group  of  domain 
experts  are  asked  to  perform  the  identical  identification,  classification,  and  grouping 
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procedure  with  the  goal  of  correlating  specific  behavior(s)  to  the  events.  The  results  are 
then  compared  and  analyzed  for  behavior/event  retention  purposes.  Concurrently,  the 
latter  group  assigns  efficacy  ratings  (point-scales)  with  respect  to  the  behavior  measures 
of  performance.  Typically,  the  developers  agree  on  an  acceptable  level  of  variance  in  the 
behavior’s  MoPs  criterion,  a  statistical  analysis  is  performed  on  the  feedback,  and  those 
behaviors/events  that  are  within  the  variance  are  retained.  Finally,  retained  behaviors 
linked  to  the  specific  events  are  then  incorporated  into  a  “series  of  [horizontal]  scales 
(one  for  each  dimension)  anchored  by  the  retained  incident  [event]”  (Schwab  et  ah,  1975, 
p.  551).  Further  refinement  from  continued  BARS  utilization  will  address  practical  and 
time-related  constraints  and  limitations. 
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IV.  METHODOLOGY 


This  chapter  describes  two  separate,  but  interrelated  phases:  the  behaviorally- 
anchored  rating  scales  (BARS)  development  phase  and  the  infantry  immersion  trainer 
(IIT)  final  data  collection  phase.  BARS  were  developed  for  the  improvised  explosive 
device  (IED)  defeat  decision-making  domain.  The  BARS  development  phase  focused  on 
the  design  and  development  of  BARS  for  use  at  the  IIT  on  Marine  Corps  Base  Camp 
Pendleton  (MCBCP),  CA.  Discussion  of  the  BARS  development  phase  also  describes 
observational  data  collected  during  IIT  research  visits  to  test  the  initial  set  of  BARS. 
Additionally,  attributes  indicative  of  decision-making  proficiency  levels  were  elicited 
during  interviews  with  subject  matter  experts  (SME).  In  the  IIT  final  data  collection 
phase,  a  revised  version  of  a  subset  of  the  BARS  developed  previously  was  employed  for 
the  purpose  of  assessing  the  metrics’  utility.  The  research  received  approval  from  the 
Naval  Postgraduate  School  (NPS)  institutional  review  board  (IRB). 

A.  BEHAVIORALLY-ANCHORED  RATING  SCALES  DEVELOPMENT 

The  maneuver  squad  leader  mastery  model  (MSLMM)  developed  for  the  United 
States  Marine  Corps  Training  and  Education  Command  describes  military  decision 
making  at  the  small-unit  level  and  served  as  the  foundation  for  developing  BARS.  The 
MSLMM  is  based  on  a  model  of  small-unit  leadership  that  describes  the,  “the  general 
progression  to  mastery  as  a  maneuver  squad  leader,  including  the  progression  of 
decision-making  proficiency”  (Ross  et  ah,  2012,  p.  1).  The  MSLMM  also  incorporates 
the  Dreyfus  and  Dreyfus,  “five  stages  of  skill  acquisition”  model  of  skill  proficiency  that 
views  proficiency  as  progressing  from  novice  to  expert  (Dreyfus  &  Dreyfus,  1986,  p.  20). 

The  MSLMM  framework  is  premised  on  15  decision-making  cognitive 

competencies  and  cognitive  and  relational  skills  (CARS).  These  cognitive  competencies 

and  CARS  were  developed  from  surveys  of  non-commissioned  officers  (NCOs)  and 

commissioned  officers  who  had  experience  with  MSL  small-unit  leadership  billets  (Ross 

et  ah,  2012).  Nine  MSL  perfonnance  areas  were  also  synthesized  from  the  NCO  and 

commissioned  officer  surveys.  “These  perfonnance  areas  encapsulate  interactions 
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between  the  decision-making  cognitive  competencies  and  CARS  into  distinct  themes” 
(see  Ross  et  ah,  2012,  p.  3).  Tables  2  and  3  list  the  “tactical  skills/tactical  thinking,”  and 
“communication”  performance  areas  that  were  selected  for  their  relevance  to  the  IED- 
defeat  decision-making  domain  (Ross  et  al,  2012).  The  selected  performance  areas 
described  in  Tables  2  and  3  provide  general  descriptions  of  the  five-levels  of 
competencies.  They  do  not  include  specific  operational  training  considerations. 


Table  2.  MSLMM  Tactical  Skills/Tactical  Thinking  Perfonnance  Areas  (From  Ross 

et  al.,  2012). 


Novice 

Advanced 

Beginner 

Competent 

Proficient 

Expert 

Tactical  Skills/Tactical  Thinkin 

> 

Understands  and 

Has  a  better 

Confident  in  his 

Planning  skills 

More  focused  on 

can  employ  basic 

understanding  of 

decision-making 

have  greatly 

taking  the 

tactics  with  his 

the  capabilities  of 

abilities  and 

improved. 

perspective  of  the 

squad.  Can  pick 

the  enemy.  Still 

now  works  with 

Mentally  simulates 

enemy.  Makes 

up  cues  in  the 

requires  some 

Higher  (instead 

possible  situations, 

decisions  based  on 

environment,  but 

guidance  and 

of  just  asking 

starts  rehearsing 

the  enemy’s 

has  difficulty 

expects  approval 

for  help)  to 

and  considers  2nd 

projected  actions 

understanding  the 

from  Higher  to 

come  up  with 

and  3rd  order 

and  abilities.  Can 

meaning  of  cues 

make  certain 

solutions. 

consequences. 

quickly  identify  a 

and  events.  Not 

decisions.  Has  a 

Higher  seeks 

Has  a  better  sense/ 

problem  and 

knowledgeable  of 

better 

him  for  advice. 

reading  of  the 

employ  squads  to 

how  to  employ  and 

understanding  of 

Has  a  better 

environment.  Can 

solve  it. 

use  all  available 

the  situation/ 

understanding  of 

pick  up  on  cues 

Visualizes  the  big 

assets.  Has 

environment  and 

how  to  read 

and  anomalies  in 

picture  and 

difficulty  planning/ 

how  to  plan  and 

environmental 

time  to  change 

considers  it  when 

considering  a 

execute  a  mission. 

cues  and  events, 

mission.  Fully 

making  decisions. 

situation  from  start 

Considers 

and  what  they 

understands  the 

Paints  a  picture  for 

to  finish  including 

consequences 

mean  to  the 

implications  of  his 

higher  and  is  now 

possible 

consequences.  May 
make  quick  and 
rash  decisions. 

when  problem 
solving  but  still 
requires  some  time 
to  make  a  decision. 

mission.  Squad 
tactical  skills  are 
sharp  and  he 
knows  how  to 
employ  his 
available  assets. 

decision  making. 

pushing 
information  up 
instead  of  pulling. 
Can  work  with  a 
broad  intent. 
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Table  3.  MSLMM  Communication  Performance  Areas  (From  Ross  et  al.,  2012). 


Novice 

Advanced 

Beginner 

Competent 

Proficient 

Expert 

Co 

mmunication 

> 

Has  difficulty 

Can  communicate 

Begins  to  adapt 

Now 

Knows  how  to 

gauging  the 

guidance  to  squad, 

communication 

communicates  with 

communicate  with 

appropriate  amount 

but  it  is  word-for- 

styles  based  on 

Platoon 

Higher  and  squad 

of  information  to 

word  what  the 

individuals. 

Commander  only 

effectively. 

report  to  Higher. 

Platoon 

Still  requires 

for  big  stuff, 

Verbalizes 

Reports  may  have 

Commander  said. 

some  guidance 

including  assets  he 

battlefield  so 

too  much 

Can  convey 

from  the  Platoon 

needs  or  situations 

others  can  see. 

information  or 

knowledge  to 

Commander,  but 

that  have  strategic 

Clearly 

leave  out  important 

Marines.  Is  able  to 

also  pulls 

implications.  Can 

communicated 

details. 

speak  more 

information. 

paint  a  picture  of 

intentions  and 

Communicates 

efficiently  with 

Talks  to  Platoon 

the  situation  that 

plans  to  squad  that 

aggressively  with 
squad  by  yelling  or 
highlighting  the 
negative  (models  a 
Drill  Instructor). 

Senior  people. 

Commander 
about  how  to 
carry  out 
mission. 
Effectively  gets 
thoughts  out 
across  the  teams 
and  can  explain 
situations 
accurately. 

the  Marines 
understand. 

they  can  work 
with.  Feels 
comfortable 
communicating 
with  Higher  and 
does  it  with  ease. 
Encourages  cross 
talk. 

BARS  developed  for  this  research  incorporated  components  of  the  MSLMM, 
information  gleaned  from  a  cognitive  task  analysis  (CTA)  on  IED-defeat  techniques 
(Ross  et  al,  2012),  IIT  operational  training  considerations,  and  SME-defined  decision¬ 
making  proficiency  stratifications.  Initially,  BARS  proficiency  levels  were  designed  to 
reflect  the  five-stages  of  the  Dreyfus  and  Dreyfus  skill  acquisition  model  (Schwab  et  al., 
1975).  However,  IED-defeat  behaviors  for  which  BARS  were  being  developed  did  not 
lend  themselves  to  producing  scales  with  five  levels  of  proficiency.  Deriving  five  levels 
of  proficiency  that  could  be  discriminated  during  the  fast-paced  scenarios  was  not 
practicable.  Future  references  to  BARS  will  be  specific  to  the  improvised  explosive 
device  (IED)  domain  and  the  IED-defeat  decision  cycles  (IE3DC)  decision-making 
model  explained  next. 

1.  IED-Defeat  Decision  Cycles  Model 

Table  4  lists  the  decision-making  fundamentals  (objectives)  in  the  IED-defeat 
domain  (predict,  prevent,  detect,  avoid,  neutralize,  and  protect).  These  fundamentals  are 
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incorporated  into  the  IED-defeat  decision  cycles  (IE3DC)  model  (Ross,  Phillips,  Moon, 
Baxter,  &  Cooper,  2008).  “The  IED3C  Model  was  produced  as  a  result  of  this  effort 
[(concept  mapping)],  in  order  to  provide  an  overarching  view  of  the  IED-defeat  task  from 
the  perspective  of  a  junior  Marine,  using  the  IED-defeat  fundamentals  tenninology 
already  in  place  as  a  construct  for  USMC  training”  (Ross  et  ah,  2008,  p.  47). 

Table  4.  Six  Fundamentals  of  the  IE3DC  model  (From  Ross  et  ah,  2008,  p.  48). 

PREDICT  the  actions  and  circumstances  of  the  IED  process. 

PREVENT  IEDs  from  being  emplaced. 

DETECT  IED-related  activities  and  IEDs  before  detonation. 

AVOID  IEDs  in  order  to  deny  the  enemy  a  target. 

NEUTRALIZE  by  reducing  or  overcoming  IEDs. 

PROTECT  the  force  against  effects  of  IEDs. 


As  shown  in  Figure  5,  the  IE3DC  model  is  premised  upon  a  concept  map 
structure  (Ross  et  ah,  2008).  Concept  maps  are  schematic  diagrams  representing 
knowledge  and  meaning  through  organized  conceptual  relationships.  Fundamentals  of 
decision  making  are  depicted  in  the  concept  map  as  nodes.  Each  node  propositionally 
links  one  fundamental  of  decision  making  to  another  in  an  IED  scenario.  The  concept 
map  prioritizes  higher-ranking  fundamentals  at  the  top  of  the  hierarchy  (protect  and 
predict)  over  lower-ranking  fundamentals  at  the  bottom  of  the  diagram  (avoid  and 
neutralize)  (Ross  et  ah,  2008). 
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Figure  5.  The  Concept  Map  of  IE3DC  model  (From  Ross  et  al.,  2008,  p.  49). 

In  the  IE3DC  concept  map,  dependency  relationships  are  depicted  as  arrows 
linking  individual  decision  making  to  “processes  related  to  one  activity  [having]  an 
impact  on  another  activity”  (Ross  et  ah,  2008,  p.  49).  The  nodal  fundamentals  provide 
the  necessary  skeletal  composition  of  the  decision-maker’s  cognitive  demands  specific  to 
the  IED-defeat  demands.  Ten  cognitive  demands  were  initially  selected  for  their 
relevance  to  the  dismounted  urban  environment  (see  Table  1,  in  literature  review 
chapter).  These  cognitive  demands  were  also  selected  based  on  the  potential  to  observe 
decision  making  associated  behaviors  without  actively  engaging  with  the  trainees. 

2.  Selected  Cognitive  Demands 

Ten  selected  cognitive  demands  that  were  used  to  develop  the  initial  set  of  BARS 
were  drawn  from  the  IED-defeat  domain  -  protect,  predict,  prevent,  detect,  avoid,  and 
neutralize  as  delineated  in  the  MSLMM  (Ross  et  al.,  2008).  For  example,  assessing 
normalcy  in  the  environment  is  a  foundational  premise  of  the  predict  activity.  An 
individual  demonstrating  mastery  assessing  normalcy  would  demonstrate  exemplary 
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behavior  regarding  identifying  anomalies  in  the  environment  (detect),  and  determine  a 
suitable  plan  to  mitigate  the  potential  adverse  impacts  on  the  unit’s  mission  and  collective 
welfare  (prevent)  (Ross  et  ah,  2008).  In  response,  an  expert  in  this  domain  would 
proactively  and  accurately  adjust  their  tactics,  techniques  and  procedures  (TTPs)  and 
activity,  and  security  posture  to  avoid  or  marginalize  the  threat  activity.  An  expert  would 
recognize  and  comprehend  potential  second-  and  third-order  effects  of  their  decisions  and 
base  their  response  on  the  logical  course  of  actions  (COAs)  from  the  enemy  perspective 
(neutralize)  (Ross  et  al,  2012). 

3.  Assessment  Factors 

a.  Verbal  and  Non-verbal  Communication 

A  military  leader  communicates  verbally  and  non-verbally  in  several  ways. 
Examples  of  verbal  and  non-verbal  communication  include  issuing  orders  and  using  hand 
signals  directed  at  other  unit  members,  sending  radio  messages  to  external  agencies  (IED 
reports  to  higher  headquarters),  and  mission-related  interactions  with  role  players  (key 
leader  engagements).  Communication  mechanics  and  interaction  techniques  should  be 
assessed  by  the  unit  leader’s  ability  to  influence  the  target’s  receptivity  and  comprehension 
of  the  message  (Mueller-Hanson  et  al.,  2007).  The  format,  brevity,  and  clarity  of  the 
infonnation  that  is  transmitted  are  paramount.  Enviromnental  and  interpersonal 
impediments  which  impact  communication  include  the  emotional  states  of  personnel, 
noise,  visual/auditory  range,  and  context.  Message  comprehension  must  be  evaluated 
according  to  the  receiver’s  ability  to  perceive  it,  and  interpret  it.  Then  the  message 
response  needs  to  be  in  a  rational  and  proportional  manner  (Mueller-Hanson  et  al.,  2007). 
In  order  to  develop  observable  communication  attributes  associated  with  three  levels  of 
decision-making  proficiency  (novice  to  expert)  interviews  were  conducted  with  SMEs. 

b.  Tactical,  Technical,  and  Procedural  Execution 

Small-unit  leaders  demonstrate  effective  leadership  by  appropriately 

responding  to  environmental  cues  and  ensuring  actions  are  aligned  with  higher  echelon’s 

intent  (Yeakey,  2002).  The  small-unit  leader’s  action  must  not  only  comply  with  the 

constraints  provided  by  their  higher  echelon  (rules  of  engagement,  ethical  behavior,  etc.), 
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they  must  cognitively  align  with  the  larger  operational  schemes  and  objectives.  A  Marine 
squad  leader’s  decision  making  is  based  on  timeliness  and  accuracy  (Goodson,  McGee,  & 
Cashman,  1989).  It  could  be  extrapolated  that  a  poor  decision  made  hastily  could  have 
the  same  adverse  effects  as  not  making  a  decision  when  conditions  warrant  it.  Moreover, 
the  BARS  developed  for  the  IIT  training  environment  also  assessed  procedural 
compliance  to  specific  safety  and  standardization  formats  (Determine  How  to  Establish 
Security  and  Cordons  (Detonated  or  non-detonated  IEDs))  (Ross  et  al.,  2008). 

c.  Location  and  Proximity  to  the  Point  of  Friction 

The  first  two  assessment  factors,  verbal  and  non-verbal  communication, 
and  TTP  execution  relate  to  the  small-unit  leaders’  explicit  actions.  These  two 
assessment  factors  do  not  address  the  enabling  decision  of  the  small-unit  leader  to 
position  himself  within  the  squad  so  he  can  effectively  communicate  (verbal  and  non¬ 
verbal  communication)  and  tactically  maneuver  the  squad  in  the  environment.  Small-unit 
leaders  are  expected  to:  (1)  have  the  basic  ability  to  perceive  changes  in  the  environment, 
(2)  know  how  to  effectively  communicate  with  their  subordinates  as  well  as  the  adjacent 
higher  echelon,  and  (3)  know  how  to  employ  their  resources. 

A  Marine  squad  is  analogous  to  a  system-of-systems.  The  unit  has 
thirteen  distinct  elements  who  work  in  three  separate,  interdependent  teams  to  achieve  a 
coordinated  goal.  The  Marine  squad  leader  is  the  squad’s  central  decision  and  execution 
node  through  which  all  inputs  and  outputs  are  processed.  The  fire  team  leaders  act  as 
semi-autonomous  responders,  feeding  stimuli  and  exercising  decisions  from  the  central 
node.  Individual  members  of  the  squad  act  as  interfaces,  interacting  with  and 
manipulating  the  environment  in  accordance  with  the  fire  team  and  squad  leader’s 
intentions.  Execution  of  the  decision  process  occurs  most  rapidly  and  correctly  when  the 
fire  team  leaders  and  individual  squad  members  are  in  communication  range  and 
proximity  to  the  Marine  squad  leader. 

When  the  squad  encounters  a  point  of  friction  it  impacts  the  unit’s 
effectiveness.  Without  central  leadership  which  is  inherent  in  the  Marine’s  squad  leader 
billet,  all  three  separate  teams  can  lose  cohesiveness  and  unity.  The  Marine  squad  leader 
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focuses  the  fire  teams  and  the  individual  members  of  the  squad  with  the  commander’s 
intent  and  purpose.  When  the  Marine  squad  leader  is  positioned  at  the  point  of  friction, 
his  decision  execution  cycle  becomes  more  efficient  because  he  has  first-hand  knowledge 
of  the  situation.  Conversely,  the  farther  away  the  small-unit  leader  is  from  that  friction 
point,  the  longer  it  will  take  to  respond  accordingly.  To  interrupt  or  attempt  to  influence 
the  squad’s  decision-making  process  can  paralyze  and  potentially  compromise  the  orderly 
function  of  the  squad’s  mission.  For  example,  in  a  dismounted  patrol  there  may  be 
circumstances  where  the  fire  team  elements  are  visually  or  geographically  separated 
(separate  alley  way  or  road)  from  the  Marine  squad  leader.  The  Marine  squad  leader  may 
have  difficulty  effectively  communicating  or  conducting  squad-level  TTPs  if  the 
separated  elements  are  simultaneously  engaged  in  kinetics  actions.  Additionally,  the 
Marine  squad  leader  could  be  right  at  the  point  of  friction  but  encounter  something 
entirely  novel  and  may  not  know  how  to  respond. 

4.  Behaviorally  Anchored  Rating  Scales  Development 

Developing  BARS  requires  a  structure  that  allows  modular  adjustments  to  the 
metric  by  accounting  for  the  variety  of  potential  scenarios  (Phillips  et  ah,  2006A). 
Initially  where  refinements  and  modifications  were  incorporated,  the  development 
process  was  envisioned  to  be  evolutionary  in  nature  so  it  would  mature.  The  following 
are  the  proposed  three  BARS  versions:  post-initial  research  visit,  post-secondary  research 
visit,  and  post-SME  interview.  The  final  BARS  would  be  consistent  in  format  and 
content  with  the  post-SME  interview  version. 

a.  Initial  Research  Visit  to  the  Infantry  Immersion  Trainer 

An  initial  visit  to  the  IIT  was  conducted  in  order  to  observe  and  become 
informed  about  the  IIT  training  and  operations.  The  IIT  staff  provided  an  operational 
overview  describing  the  current  IIT  training,  assessment  methodology,  technology,  and 
training  participants.  Observations  of  Marine  small-units  conducting  operational  training 
with  semi-structured  scenarios  were  conducted  in  both  indoor  and  outdoor  training 
ranges.  A  brief  was  provided  to  the  IIT  staff  on  this  thesis  research  background  and 
purpose,  and  addressed  how  the  BARS  would  provide  a  more  objective  and  quantifiable 
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assessment  of  trainees’  proficiency.  The  majority  of  the  feedback  from  the  IIT  staff 
indicated  that  any  assessment  techniques  would  have  to  be  easy  to  use  and  quick  to 
implement.  Moreover,  any  assessment  would  need  to  have  a  negligible  impact  on  the 
trainer’s  attention  and  interactions  with  the  training  audience. 

IIT  operational  training  is  designed  and  developed  around  a  particular 
operational  scenario  and/or  set  of  mission  essential  tasks.  IIT  scenario  development  is 
typically  illustrated  through  a  storyboard.  This  storyboard  fonnat  facilitates  the  IIT  staff 
in  organizing  the  scenario  execution  and  narrative.  A  storyboard  delineates  pivotal 
milestones  within  the  scenario  that  a  Marine  small-unit  will  encounter.  As  depicted  in 
Figure  6,  the  information  at  the  top  of  the  scenario  storyboard  describes  the  unit  (1/7), 
training  day  (TD-1),  training  facility  (indoor),  scenario  number  (1),  and  description  (QRF 
(quick  reaction  force)  to  ANSF  (Afghan  national  security  forces)  IED  Find  at  the  Sqd 
(squad)  level).  Along  the  left-hand  column  tactical  objectives,  atmospherics,  visuals 
(pyrotechnics),  and  disposition/composition  of  the  enemy  role-players  (enemy  activity) 
are  described.  A  graphical  depiction  of  the  expected  scenario  execution  is  also  presented. 
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Mission:  Link  up  with  ANSF 
patrol  IVO  of  the  market 
IOT  assist  them  in  securing 
the  area  around  an  IED 
that  they  have  identified. 

Atmospherics:  ANSF 
having  trouble  keeping 
locals  clear  of  IED. 

Visuals:  IED  covered  with 
disturbed  earth  in  the 
market. 

-Pressure  plates  IVO  1A 
and  3B  IEDs. 

Enemy  Activity: 

-1A  and/or  3B  IEDs 
detonate  if  pressure  plates 
are  stepped  on. 


1/7  TD-1,  Indoor  Scenario  1:  QRF  To  ANSF  IED  Find  (Sqd  Level) 


Figure  6.  IIT  Scenario  1  Storyboard:  QRF  to  ANSF  IED  Find  (From  Fennell,  2013B). 
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The  IIT  scenario  storyboard  is  an  internally  distributed  planning  and 
operations  document.  The  IIT  staff  can  change  the  specifics  of  the  scenario  prior  to  and 
during  scenario  execution  to  meet  the  training  requirements  of  a  particular  unit.  The 
training  staff  will  only  disclose  general  scenario  atmospherics  and  tactical  objective(s)  to 
the  training  participants.  The  IIT  staff  adjusts  scenario  dynamics  and  atmospherics  as 
needed  to  adapt  to  the  squad’s  level  of  proficiency  and  to  achieve  specific  learning 
objectives. 


b.  Behaviorally  Anchored  Rating  Scales  Initial  Development 

The  goal  for  this  phase  of  the  research  was  to  synthesize  the  existing  body 
of  knowledge  on  BARS  development  and  develop  an  initial  set  of  BARS.  Table  5  lists 
and  describes  the  ten  cognitive  demands  selected  from  the  IE3DC  model,  and  provides 
descriptions  of  IED  related  behavioral  indicators,  conditions,  and  cues. 
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Table  5.  IE3DC  Model  Cognitive  Demands  and  Descriptions 
(From  Ross  et  al.,  2008,  pp.  51-56). 


Cognitive  Demands 

Descriptions 

1.  Assess  Normalcy  in  the 
Environment 

Establishing  a  baseline  understanding  of  the  environment  is  one 
of  the  pillars  on  which  all  IED-defeat  decisions  rest. 
Knowledge  of  what  is  normal  -  from  behaviors  of  the  populace 
to  placement  of  specific  objects  in  the  operating  environment  - 
enables  detection  of  change  and  identification  of  anomalies  that 
may  be  indicative  of  insurgent  signatures  and  devices  in  the 
process  of  emplacement.  Normalcy  may  be  passed  from  unit  to 
unit  during  Reliefs  in  Place,  from  leadership  to  platoon  during 
battle  space  preparation,  and  from  Marine  to  Marine  during  the 
course  of  patrols  and  convoys. 

2.  Assess  Probable  IED 
Locations 

In  addition  to  assessing  when  IED  attacks  are  likely,  Marines 
continuously  assess  where  attacks  may  occur.  These 

assessments  occur  at  a  general  level  -  i.e.,  knowing  insurgent 
operation  areas  and  trends  in  behavior  -  and  at  a  specific  level 
-  i.e.,  knowing  hot  spots  and  evaluating  the  features  of  the 
environment  that  favor  enemy  activities.  In  conjunction  with 
assessing  where  an  IED  may  be  located,  another  component  of 
this  judgment  is  where  particular  types  of  IEDs  are  likely  to  be 
used.  For  example,  high-yield  IEDs  such  as  [explosively 
formed  penetrators]  EFPs  are  more  typically  used  on  rural 
routes,  while  lower-yield  IEDs  can  be  anticipated  more 
frequently  on  urban  routes.  Marines’  assessments  of  probable 
IED  locations  have  a  direct  impact  on  route  planning  activities 
as  well  as  on-the-fly  decisions  (i.e.,  during  a  patrol  or  convoy) 
regarding  which  routes  to  traverse. 

3.  Notice  Indicators  of  a 
Dangerous  Situation 
(Terrain  or  People) 

This  decision  reflects  how  skilled  Marines  detect  an  IED  threat 
by  focusing  not  just  on  indicators  of  a  device,  but  also  on 
indicators  throughout  the  environment  that  suggest  they  may  be 
entering  a  zone  where  an  IED  is  located.  This  decision 
necessarily  interacts  with  those  under  Predict;  Marines  use 
expectations  they  have  about  where  IEDs  might  be  located  and 
when  they  are  most  likely  to  be  emplaced  in  order  to  prime 
their  recognition  of  danger  indicators.  There  are  two  primary 
indicators  Marines  assess  when  noticing  a  dangerous  situation  - 
terrain,  and  people  and  their  activities.  Terrain  features 
include:  cues  on  the  road  -  e.g.,  potholes;  tread  patterns  -  and 
cues  around  the  road  -  e.g.,  structures  such  as  bridges  and 
abutments.  People  and  activity  features  include:  the  extent  of 
current  and  recent  traffic;  the  absence  of  civilians  and  children; 
and  movement  in  buildings. 

Continued  on  next  page 
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4.  Notice  Indicators  of  an 
IED  Device 

Another  pillar  of  IED  defeat  lies  in  noticing  the  indicators  of  a 
device.  This  includes  noticing  components  of  a  device  in  the 
context  of  vehicle  and  home  searches,  and  identifying  IEDs 
that  are  emplaced  by  their  component  parts.  Exhaustion  and 
complacency  are  common  challenges  associated  with 
identifying  IEDs  in  the  environment.  However,  when  a 
Marine’s  attention  is  focused  he  can  anticipate  an  IED  attack, 
or  when  his  attention  is  directed  to  high-threat  areas  as  a  result 
of  his  experience,  detection  becomes  more  manageable.  An 
extensive  list  of  cues  and  factors  involved  in  this  judgment  are 
provided  in  the  cognitive  demand  tables  (CDTs).  As  the 
USMC  well  knows,  these  cues  and  factors  are  only  as  current 
as  the  improvisations  of  the  enemy. 

5.  Assess  the  Nature  and 
Extent  of  All  Current 
Threats 

This  decision  concerns  the  devices  as  well  as  other  insurgent 
activities  that  surround  them.  Marines  must  simultaneously 
judge  what  they  have  detected  and  what  threat  it  poses,  and 
what  other  threats  may  be  posed  by  other  devices  or  forms  of 
attack.  These  judgments  consider  whether  devices  may  be 
hoaxes,  whether  secondary  devices  may  exist,  and  the 
possibility  of  ambush  -  any  of  which  is  possible  in  any 
situation.  Marines  consider  information  available  about  the 
immediate  scene,  the  protective  equipment  they  have  in  place, 
their  vulnerability,  and  even  the  nature  of  reporting  surrounding 
a  device. 

6.  Determine  How  to 

Trace  to  the  Point  of 
Origin 

For  devices  that  suggest  a  command  wire-initiated  device, 
Marines  may  attempt  to  trace  elements  of  the  device  to  the 
Point  of  Origin,  or  POO,  site.  Cues  considered  are  wires  and 
other  elements.  Tracing  is  made  challenging  by  the  potential 
for  booby  traps  and  secondary  devices,  and  because  some  POO 
sites  may  be  great  distances  from  the  point  of  detonation. 
Marines  must  judge  the  risk  associated  with  tracing  the  wire. 
When  risk  is  deemed  high,  Marines  may  visually  attempt  to 
trace  a  site,  and/or  walk  identifiable  paths  to  the  POO  site,  a 
strategy  that  amplifies  the  challenges. 

7.  Identify  and  Respond 
to  Triggerman 
(Detonated  or  non- 
detonated  IEDs) 

For  devices  that  require  command  detonation,  Marines  look  for 
triggermen.  The  search  may  include  looking  for  a  line  of  sight 
and  places  that  are  concealing,  and  typically  goes  hand-in-hand 
with  searching  for  the  Point  of  Origin  site.  In  cases  where  a 
triggerman  is  identified.  Marines  must  then  consider  how  to 
deal  with  him.  Options  here  include  giving  chase,  firing,  or 
doing  nothing,  and  each  of  these  options  may  be  considered 
with  regard  to  safety,  feasibility  of  success,  and  security. 

Continued  on  next  page 
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8.  Determine  How  To 
Establish  Security  and 
Cordons  (Detonated  or 
non-detonated  IEDs 

The  goal  of  this  decision  is  one  of  safety  and  of  maintaining  the 
integrity  of  the  environment  around  a  detected  IED.  Marines 
want  to  keep  everything  within  the  cordon  that  was  there  at 
detection,  and  keep  anything  else  out.  In  determining  when  to 
establish  a  cordon,  Marines  are  challenged  by  the  often  lengthy 
duration  of  time  EOD  [(explosive  ordinance  disposal)]  may  take 
to  arrive  on  scene.  In  determining  how  to  establish  a  cordon, 
Marines  must  take  into  account  the  terrain  around  the  IED  site, 
manpower,  and  vehicles  at  their  disposal. 

9.  Consider  How  to 
Respond  Immediately 
to  Detonation 
(Detonated  IEDs) 

Upon  detonation,  Marines  must  make  split-second  decisions 
about  how  to  respond.  These  decisions  include  whether  to 
approach  the  detonation  site,  whether  to  dismount,  and  where 
to  place  vehicles  in  a  convoy.  All  of  the  other  Neutralize 
decisions  and  judgments  above  come  immediately  into  play, 
and  are  made  all  the  more  challenging  by  the  stress  of  enduring 
the  detonation  and  deep-seated  first  impulses  to  help  victims 
and  fight.  Platoon  leadership  must  also  consider  when  and  how 
to  communicate  with  higher  headquarters.  Other  post¬ 
detonation  decisions  include  assessing  the  damage,  responding 
to  injuries  or  casualties,  and  responding  to  disabled  vehicles. 
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The  initial  version  of  the  BARS  was  based  on  the  ten  cognitive  demands 
listed  and  described  in  Table  1  of  the  literature  review  chapter.  As  depicted  in  Figure  7, 
the  BARS  format  consisted  of  three  parts.  Each  part  was  annotated  by  the  part  number  (I, 
II,  and  III)  in  bold  red  lettering.  Part  I  stated  the  general  scenario  description 
(unexploded  IED),  and  the  associated  cognitive  decision  related  to  the  perceptual  task 
(notice  indicators  of  an  IED  device).  In  addition  to  the  cognitive  decision,  part  I  also 
specified  the  perceptual  task  related  to  the  IED  (identify  IEDs).  Part  II  is  a  5-point  scale 
based  on  the  Dreyfus  and  Dreyfus  skill  acquisition  model  where  proficiency  ranges  from 
novice  to  expert  (Dreyfus  &  Dreyfus,  1986).  The  scales  were  intended  to  be  used  by  the 
assessor  to  observe  the  Marine  squad  leader’s  behavior,  and  then  mark  the  proficiency 
level  associated  with  the  cognitive  decision  in  part  I,  using  the  related  descriptors  of 
observable  behaviors  annotated  in  Part  III.  Part  III  consisted  of  multiple  behavior  factors 
(listed  across  the  rows),  containing  five  levels  of  observable  cognitive  linked  behaviors 
(columns). 

The  central  tenet  of  the  BARS  utility  was  the  ability  to  associate 
observable  behaviors  demonstrated  by  the  training  participants  with  objective  criterion 
that  differentiated  between  highly  proficient  (expert)  and  novice  behavior.  In  addition  a 
report  on  CTA  was  used  to  develop  initial  descriptors  (Ross  et  al,  2012).  However, 
observable  indicators  of  the  decision-making  behavior  were  not  found  in  the  report  due  to 
an  incompatibility  between  the  scope  of  the  CTA  on  IED-defeat  performance  and  the  IIT- 
specific  training.  The  existing  literature  provided  infonnation  to  develop  the  set  of 
behavioral  cues  for  the  BARS.  As  depicted  in  Figure  7,  the  cognitive  demand  (notice 
indicators  of  an  IED  device)  includes  observable  decision-making  behaviors  (identifiable 
cues  of  IED  danger  areas,  IED  assessment  techniques,  and  cultural  and  environmental 
IED  emplacement  knowledge).  During  the  researcher’s  second  visit  to  the  IIT  an  initial 
version  of  BARS  were  employed  to  test  the  BARS  utility. 


47 


Scenario:  Unexploded  IED 


Decision:  Notice  Indicators  of  an  IED  Device  (Identify  lEDs) 
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Figure  7.  Example  of  an  Initial  Behaviorally- Anchored  Rating  Scale:  Notice  Indicators 

of  an  IED  Device. 


c.  Second  Research  Visit  to  Infantry  Immersion  Trainer 

The  purpose  of  the  second  visit  to  the  IIT  was  to  further  develop  the 
researcher’s  familiarity  regarding  IIT  operations,  training  and  scenario  development,  and 
BARS  utilization.  The  first  day  focused  on  familiarization  with  IIT  operations,  and 
scenario  development  and  execution.  During  the  following  three  training  observation 
days,  the  focus  was  on  acclimatizing  to  the  IIT  training  environment  and  utilization  of  the 
metrics  already  developed.  The  following  list  includes  the  research  objectives  for  the 
second  visit:  (1)  To  achieve  a  more  comprehensive  understanding  of  the  IIT  staff  roles 
and  responsibilities;  (2)  better  understand  the  IIT  training  scenario  development  and 
execution  process;  (3)  comprehend  the  limitations  and  constraints  of  the  IIT  staff  and 
facilities;  (4)  elicit  feedback  from  the  IIT  staff  regarding  thesis  research  and  BARS 
development;  and  (5)  coordinate  follow-on  final  data  collection. 
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The  following  observations  occurred  over  a  four-day  period.  Each  day 
provided  additional  insights  and  lessons  learned. 

(1)  Day  One  Observations.  Upon  arrival  at  the  MCBCP  IIT,  the 
IIT  director  and  operations  officer  provided  an  overview  of  IIT  operations  ranging  from 
staffing  and  training  issues,  to  facilities  and  contractor  support  considerations. 
Additionally,  information  was  provided  regarding  first  Marine  Expeditionary  Force  (I 
MEF)  IIT  utilization  policy,  and  ongoing  experimentation  and  assessment  projects.  In 
subsequent  conversations,  video  and  audio  capturing  technologies  were  discussed  as  a 
means  to  facilitate  after-action  reviews  (AARs)  and  trainer  assessments. 

Between  the  first  two  IIT  visits,  a  change  occurred  in  the  manner 
in  which  the  IIT  staff  captured  and  integrated  selected  scenario  video  segments  into  the 
AAR.  The  legacy  instrumented  video/audio  capturing  system  called  TVCS  (tactical 
video  capture  system),  had  an  inherent  and  significant  delay  in  processing  selected  videos 
for  the  AAR.  Given  that  training  scenarios  typically  required  30-60  minutes  to  complete, 
time  spent  processing  TVCS  videos  for  the  AAR  had  become  too  time-consuming  and 
ineffective  as  the  primary  debriefing  aid.  The  IIT  staff  implemented  a  change  from  the 
instrumented  close-circuited  TVCS  to  a  hand-held  contour  camera  system  employed  by 
individual  IIT  staff.  Usage  of  handheld  cameras  reduced  the  inherent  processing  delay  in 
the  TCVS/Spotlight  systems  from  20  minutes  to  5  minutes.  This  improvement  allowed 
the  IIT  staff  to  increase  the  number  of  training  scenario  runs  significantly  over  a  four-day 
training  evolution. 

In  additional  conversations,  the  IIT  operations  officer  elaborated 
on  their  training  philosophy  regarding  scenario  development,  where  he  emphasized  a 
winnable,  applicable,  and  coachable  approach  to  training  Marines.  Each  event  must 
afford  training  participants  a  modicum  of  mission  success  and  personal  growth.  Salient 
tactical  and  technical  execution  points  must  demonstrate  relevance  and  applicability  to 
preparing  the  training  audience  for  future  operational  deployments.  Finally,  events  must 
emphasize  an  open-ended  decision-making  process,  that  is  to  say,  there  is  not  always  one 
correct  way  to  respond  to  a  given  situation.  The  training  environment  must  be  conducive 

to  and  provide  the  impetus  for  IIT  staff  guidance  and  COA  training  audiences. 
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An  example  of  the  BARS  was  provided  to  the  IIT  operations 
officer  and  he  was  asked  to  provide  feedback  regarding  the  applicability  of  this  metric. 
His  initial  feedback  focused  on  the  individual  units  and  their  respective  pre-deployment 
training  program  (PTP)  cycle.  He  stated  that  units  further  along  in  their  PTP  cycle  have  a 
distinct  and  expected  advantage  regarding  TTP  proficiency  and  execution  compared  to 
units  who  are  not  as  far  along  in  their  own  PTP  cycle.  Units  early  in  their  respective  PTP 
cycles  are  adversely  affected  by  personnel  turnover,  unit  cohesion,  and  leadership 
continuity  issues.  His  opinion  was  that  a  trainer  who  uses  BARS  would  need  to  consider 
where  the  unit  is  in  their  PTP  when  assessing  of  the  unit’s  performance. 

(2)  Day  Two  Observations.  Day  two  consisted  of  observing 
squads  engaging  in  scenario  runs  in  the  indoor  facility.  Figure  8  shows  the  AAR  Room 
which  served  as  the  main  debriefing  location  for  the  indoor  scenario  runs.  A  total  of  six 
training  scenario  runs  and/or  AARs  were  observed,  specifically  training  day  one,  indoor 
scenario  one  and  three. 

After  querying  the  unit’s  Platoon  Commander  regarding  scenario 
training  run  information,  the  Platoon  Commander  confirmed  that  the  squad  was  in  a  QRF 
role  providing  assistance  to  the  ANSF  for  securing  the  bazaar  due  to  an  identified  IED 
threat  (training  day  one,  indoor  scenario  one).  With  this  knowledge,  it  was  decided  that 
the  following  three  BARS  would  be  used:  “Notice  indicators  of  an  IED  device,” 
“Determine  how  to  establish  security  and  cordon,”  and  “Consider  how  to  respond 
immediately  to  detonation.”  The  following  paragraphs  summarize  observations  and 
highlights  of  day  two  training  scenario  runs  utilizing  the  BARS. 
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Figure  8.  Phase  I  AAR  Room  (From  Thelen,  2013). 


(3)  Training  Day  One,  Indoor  Scenario  One.  1st  Squad:  As 
depicted  in  Figure  6,  the  Marine  squad’s  primary  mission  objective  was  to  interact  with 
the  ANSF  at  a  simulated  bazaar  and  support  a  cordon  and  security  in  the  bazaar  around 
an  identified  IED.  A  secondary  objective  was  to  support  a  simulated  EOD  unit  with 
protection  and  security.  Figure  9  shows  the  Marine  squad  entering  the  indoor  training 
facility  and  moving  slowly  toward  the  bazaar  (objective  area). 
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Figure  9.  Marines  navigate  through  alleyways  of  the  Phase  I  of  IIT 

(From  Thelen,  2013). 

The  lead  fire  team  element  of  the  squad  was  spearheaded  by  a 
Marine  holding  a  compact  metal  detector  (CMD).  The  lead  Marine,  referred  to  as  the 
Sweeper,  carried  and  used  the  CMD  to  search  for  high-density  objects,  such  as  mines  or 
IEDs  in  the  intended  navigable  direction  and  path  of  the  squad’s  movement.  Once  the 
CMD  detected  a  metallic  (high-density)  object  under  its  sensor,  it  emitted  a  visual  and 
auditory  indication.  As  previously  discussed,  the  IIT  is  a  converted  former  tomato 
packing  plant  with  some  legacy  infrastructure  artifacts,  such  as  metal  rebar  throughout 
the  concrete  surface  of  the  indoor  facility.  This  particular  artifact  caused  the  CMD 
device  to  continually  emit  indications  of  highly-dense  objects.  The  IIT  staff 
accompanying  the  respective  unit  compensated  for  the  metal  artifact  by  providing  the 
Sweeper  an  audible  indication  of  the  correct  emanation.  A  verbal,  “soft,”  “weak,” 
“hard,”  or  “strong,”  indication  was  given  by  the  trainer  to  simulate  the  level  of  detection. 
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As  the  squad  approached  and  entered  the  objective  area,  the 
Marine  squad  leader  moved  forward  and  interacted  with  the  ANSF  unit  leader.  It 
appeared  that  the  squad  leader  was  receiving  a  brief  from  the  ANSF  unit  leader  regarding 
the  bazaar  danger  area.  Additionally,  the  ANSF  unit  leader  signaled  the  general 
proximity  of  the  IED  device  at  3B  (Figure  10).  With  the  general  proximity  of  the  IED 
shown,  the  Marine  squad  leader  paused  a  moment  and  then  visually  confirmed  the  ANSF 
leader’s  claim.  The  “Notice  indicators  of  an  IED  device”  BARS  (see  Figure  7)  was 
utilized  at  this  juncture  in  the  scenario. 

The  “Notice  indicators  of  an  IED  device”  BARS  consists  of  three 
factors  shown  in  Figures  10,  11,  and  12.  These  three  factors  include  noticing  danger  area 
cues,  IED  cue  detection  response  performance,  and  cultural  and  environmental 
discrimination  of  IED  emplacement  characteristics.  The  first  behavior  factor  (Figure  6), 
“Notice  IED  danger  area  cues”  was  assessed,  followed  by  the  other  two  behavior  factors 
in  order  (Figure  11  and  12).  When  observing  the  Marine  squad  leader’s  response  to  the 
ANSF  unit  leader’s  hand-and-arm  signals  and  his  subsequent  scrutinizing  the  terrain,  it 
appeared  that  the  Marine  squad  leader  was  attempting  to  perceive  and  categorize  the 
IED/mine  cues  from  the  terrain.  After  reading  the  descriptors  associated  with  each  point 
on  the  BARS  scale,  the  rater  detennined  that  the  fourth  marker,  “understands  cues  that 
indicate  danger  areas  with  IEDs,”  was  an  appropriate  rating. 

In  highlight,  assessment  of  the  Marine  squad  leader’s  behavior 
related  to  identifying  IED  danger  area  cues  was  completely  subjective  in  nature.  This 
assessment  used  length  of  time  spent  visually  looking  into  the  danger  area  as  a  positive 
indication  of  a  systematic  search  methodology.  Additionally,  the  assessment  assumed  the 
amount  of  time  the  Marine  squad  leader  spent  surveying  the  danger  area  was  positively 
correlated  to  correctly  identifying  the  IED/mine  location.  These  two  assumptions  were 
based  on  three  unsubstantiated  biases  that  influenced  the  assessment.  First,  the  Marine 
squad  leader  was  completely  receptive  to  the  ANSF  unit  leader.  This  was  unfounded 
because  there  was  not  the  ability  to  objectively  verify  the  Marine  squad  leader’s 
confidence  in  the  ANSF  unit  leader’s  knowledge.  Second,  the  Marine  squad  leader  had 
competent  expertise  to  locate  and  identify  the  IED/mine  threat  at  3B  (remote  control, 
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command-weight  switch,  command-wire,  etc.)-  This  information  about  the  Marine  squad 
leader  was  unknowable  without  retrieving  his  demographic  information.  Third,  the 
Marine  squad  leader  comprehended  the  approximate  IEDs/mine’s  effective  casualty 
radius  (ECR).  The  Marine  squad  leader’s  knowledge  and  awareness  of  the  IED 
characteristics  was  also  unknowable  without  demographic  information  on  the  Marine 
squad  leader.  These  assumptions  and  their  underlying  biases  were  not  supported  or 
substantiated  by  other  objective  evidence. 
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Figure  10.  Behaviorally- Anchored  Rating  Scale  Version  One:  Notice  Indicators  of  an 

IED  Device  (IED  Danger  Area  Cues). 


The  Marine  squad  leader’s  verbal  and  non-verbal  (hand-and-arm 
signals)  interactions  with  other  members  in  his  unit  were  observed  in  addition  to  his 
interactions  with  the  ANSF  unit  leader.  It  is  noteworthy  that  the  ability  to  hear  the 
Marine  squad  leader’s  radio  communication  from  the  Phase  I  AAR  Room  was  extremely 
difficult  given  the  limiting  acoustics  of  the  indoor  facility,  and  limited  access  to  the 
training  participant’s  radios.  The  indoor  closed-circuit  television  (CCTV)  camera 
perspective  and  AAR  room  monitor  resolution  additionally  prevented  the  rater’s  ability  to 
visually  decipher  verbal  interactions  between  the  Marines,  role-players,  and  IIT  staff 
during  training  scenario  runs.  Communication  behavior  of  the  Marine  squad  leader  could 
only  be  inferred  through  observing  his  non-verbal  communication.  After  observing  the 
Marine  squad  leader’s  communicative  behavior,  it  was  ascertained  that  the  Marine  squad 
leader’s  “IED  danger  area  response  performance”  was  consistent  with  BARS  level  four 
“competent  real-time  identifying  of  danger  areas  with  IEDs”  (Figure  11). 

In  hindsight,  the  assessment  of  this  behavior  factor  had  similar 
pitfalls  described  in  the  previous  “noticing  indicators  of  an  IED  device”  factor.  There 
was  not  a  distinct  means  of  articulating  the  timing  and/or  accuracy  of  this  behavior 
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factor’s  performance  criteria.  Additionally,  it  was  not  possible  to  observe  when  the 
Marine  squad  leader  utilized  a  reference  such  as  a  checklist  from  the  Phase  I  AAR  Room. 
Having  distinct  and  definable  response  performance  criteria  codified  prior  to  scenario 
training  run  execution  would  enhance  this  factor’s  usability. 


Req  verbatim  use  Uses,  not  req,  IED  Competent  but  Competent  real- 

|  1  IED  checklist  to  [  |  chklist  to  identify  |  |  delayed  w/  |  |  time  identifying 

identify  danger  danger  areas  w/  identifying  danger  danger  areas  w/ 

areas  w/IEDs  lEDs  areas  w/  lEDsw/o  lEDs 

IED  checklist 

Figure  11.  BARS  Version  One:  Notice  Indicators  of  an  IED  Device  (IED  Danger  Area 

Response  Performance). 

The  Marine  squad  leader  was  observed  surveying  the  bazaar 
danger  area  for  potential  IEDs.  It  was  inferred  that  the  Marine  squad  leader  was  actually 
searching  for  physical  indicators  of  IED  emplacement.  IED  emplacement  knowledge 
was  critical  given  that  IED  emplacement  characteristics  can  change  substantially  from 
one  geographic  region  to  another.  Understanding  the  geographically  specific  IED 
emplacement  characteristics  of  the  deployable  operational  environment  provides  a  greater 
situational  awareness  and  improves  self-protection.  Although  there  are  distinct 
differences  in  IEDs  used  in  different  locations,  there  are  specific  IED  heuristics  shared 
across  these  geographic  domains.  Future  reference  to  IEDs  will  be  specific  to  ground- 
borne,  stationary  devices. 

How  the  IED  is  actuated  provides  important  information  related  to 
the  IEDs  physical  and  emplacement  characteristics.  The  IIT  trainers  and  staff  provide 
limited  information  to  the  training  participants  regarding  the  simulated  IEDs  effective 
causality  ranges  (ECRs)  in  the  IIT  training  environment.  The  Marine  squad  leader’s 
ability  to  recognize  IED  emplacement  characteristics  is  important  in  determining  standoff 
ranges  and  cordon  considerations.  When  rating  the  Marine  squad  leader’s  IED 
emplacement  knowledge  and  awareness  using  the  BARS  in  Figure  12,  he  was  rated  a 
“competent”  evaluation  because  no  Marine  stepped  on  or  actuated  the  simulated  mine  in 
the  scenario. 


Anticipates  & 
locates  danger 
areas  w/  IEDs 
based  upon 
enemy  TTPs 
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Figure  12.  BARS  Version  One:  Notice  Indicators  of  an  IED  Device  (IED  Emplacement 

Knowledge  and  Awareness). 

As  the  cordon  was  quickly  established  around  the  bazaar  area  with 
the  assistance  of  the  ANSF,  it  appeared  that  the  ANSF  were  tasked  to  move  the  civilians 
(role-players)  out  of  the  bazaar  area  cordon.  The  size  of  the  simulated  bazaar  was 
distinctly  smaller  than  what  is  available  in  the  outdoor  (Phase  II)  training  range.  It 
appeared  that  it  would  now  be  appropriate  to  utilize  the  “determine  how  to  establish 
security  and  cordon”  BARS.  Figure  13  shows  this  BARS  has  four  behavior  factors 
related  to  locating  the  IED.  They  are  as  follows:  IED  location,  IED  emplacement 
knowledge  and  awareness,  5/25  meter  searches,  and  integrity  and  security  of  the  cordon 
until  EOD  arrives.  After  establishing  the  cordon  at  the  entrances  to  the  bazaar,  the 
Sweeper  was  tasked  to  clear  designated  locations  of  the  cordon  area. 

This  metric’s  first  two  behavior  factors  were  previously  discussed 
in  the  “notice  indicators  of  an  IED  device”  BARS  and  assessed  at  the  same  proficiency 
levels.  The  drawback  associated  with  nesting  several  distinct  and  independent  factors 
was  a  significant  lesson  learned  in  the  BARS  development  process.  The  next  behavior 
factor  relating  to  the  5/25  meter  searches  of  the  danger  area  was  done  independent  of  the 
Marine  squad  leader’s  direction.  Observing  whether  the  Marine  Squad  leader 
individually  conducted  5/25  meter  searches  was  impossible  from  the  vantage  point  of  the 
Phase  I  AAR  Room.  The  only  indication  of  this  behavior’s  proficiency  was  the  number 
of  potential  IEDs  discovered  and  false  positives.  Since  this  information  was  not 
technically  known  prior  to  scenario  execution,  nor  incorporated  into  the  existing  metric, 
this  behavior  factor  was  not  assessed. 
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Scenario:  Unexploded  IED 


Decision:  Determine  How  to  Establish  Security  and  Cordons 
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Figure  13.  BARS  Version  One:  Determine  How  to  Establish  Security  and  Cordons. 


The  squad’s  Platoon  Commander  was  also  observing  the  scenario 
execution  from  the  Phase  I  AAR  Room  and  was  role-playing  the  simulated  combat 
operations  center  (COC)  (squad’s  higher  headquarters).  The  COC  was  not  receiving 
radio  communication  from  the  Marine  squad  leader  regarding  the  identified  IED.  The 
Platoon  Commander  commented  aloud  that  the  lack  of  radio  communication  in  the  form 
of  an  IED  9-Line  report  was  not  normal.  This  radio  communication  irregularity  was  an 
issue  that  in  hindsight  plagued  future  training  scenario  runs.  The  issue  was  identified  and 
attributed  to  two  factors.  The  first  factor  related  to  an  improper  radio  crypto  logic  fill, 
and  the  second  factor  related  to  the  internal  infrastructure  of  the  Phase  I  facility.  Both 
issues  separately  could  cause  a  severe  impact  to  sustaining  communication  and  both 
factors  together  would  make  radio  communication  nearly  impossible. 

Radio  communication  was  successfully  established  after  the 
Platoon  Commander  installed  the  proper  radio  crypto  logic  fill  and  adjusted  his 
transmission  location  and  antenna  configuration.  When  the  Marine  squad  leader 
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successfully  communicated  the  IED  9-line  report  it  was  in  an  incorrect  format  and  the 
report’s  contents  were  not  relevant  to  the  IIT  scenario.  The  Marine  squad  leader  did  not 
provide  the  IIT  specific  building  location  and  the  radio  communication  cadence  was 
erratic.  This  issue  forced  the  Platoon  Commander  to  explain  over  the  radio  the  correct 
reporting  fonnat,  creating  friction  and  delay  to  the  unfolding  scenario.  Additionally,  the 
COC  had  to  direct  the  Marine  squad  leader  to  search  and  report  the  status  of  other 
potential  IEDs  in  the  cordon  area.  In  hindsight,  these  types  of  issues  occurred  repeatedly 
throughout  the  training  day,  and  were  not  mutually  exclusive  to  this  squad.  Finally,  the 
simulated  EOD  unit  (IIT  staff)  was  brought  into  the  scenario,  the  EOD  disarmed  the  IED 
and  the  scenario  came  to  an  end. 

The  final  BARS  behavior  factor,  “integrity  and  security  of  the 
cordon  until  EOD  arrives”  was  assessed.  As  depicted  in  Figure  13,  the  behaviors 
included  in  this  behavior  factor  were  the  exclusive  control  of  the  cordon  with  respect  to 
avenues  of  approach,  the  internal  quarantine  of  IED/mine  devices  (location, 
identification,  and  marking),  and  finally  the  EOD  link-up  plan  and  execution.  The 
scenario  did  possess  some  artificiality  with  respect  to  the  trainer’s  compression  of  the 
EOD  timeline.  The  amount  of  time  required  by  the  simulated  EOD  to  arrive,  get  briefed 
and  finally  neutralize  the  known  IED  was  relatively  fast  (<30  minutes).  Conversely,  the 
time  required  in  an  operational  environment  for  EOD  to  arrive  after  receiving  the  request, 
receiving  a  proper  briefing,  and  then  disarm  the  unexploded  ordinance  (UXO)  would 
requires  substantially  more  time  than  what  was  available  in  the  scenario  timeline  (indoor 
scenarios  usually  last  approximately  45  minutes  to  one  hour).  This  artificial  time 
consideration  was  not  meant  to  question  the  relevance  of  the  cordon  actions,  rather  the 
truncated  time  line  forced  the  Marine  squad  leader  to  ensure  his  decisions  were  made  in 
an  efficient  manner.  Observing  the  EOD/Marine  squad  leader’s  interaction  over  a  ten 
minute  period,  the  rater  assessed  that  he  competently  executed  the  behaviors  at  level 
three. 

With  the  IED/mine  disarmed,  the  scenario  training  was  complete 
and  the  squad  moved  to  the  Phase  I  AAR  Room.  The  trainer  began  the  AAR  by 
explaining  the  video’s  perspective  shown  on  the  monitor  would  be  from  the  point  of  view 
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of  a  potential  adversary  or  threat.  The  trainer  emphasized  that  the  squad  should 
comprehend  the  enemy’s  perspective  in  dismounted  environments.  He  continued  by 
stating  the  Marines  needed  to  appreciate  their  adversaries  as  active  components  of  the 
battle  space.  The  trainer  subsequently  presented  video  excerpts  of  the  squad  as  they 
progressed  through  the  scenario  training  run.  Each  pause  of  the  video  focused  on 
individual  and  collective  deviations  (learning  points)  related  to  internal/external 
communication,  roles  and  responsibilities,  TTP  conduct  and  execution,  and 
cordon/security  conduct  and  procedures.  The  trainer’s  point  was  to  elicit  individual 
perspectives  of  the  situation,  present  the  actual  ground  truth  of  the  event,  and  contrast  the 
training  participant’s  perspectives  against  the  actual  transpired  events. 

Training  Day  One,  Indoor  Scenario  One — 2nd  Squad:  The 

second  scenario  run  was  significantly  different  from  the  first  scenario  run  due  to  a  Marine 
becoming  a  casualty  as  a  result  of  stepping  on  a  simulated  pressure -plate  mine  between 
A5  and  A3  (see  Figure  6).  Two  casualties  occurred  from  the  detonation  in  this  scenario, 
a  Marine  and  an  ANSF  soldier.  Both  casualties  were  dragged  into  the  mock  police 
station  compound  located  at  A5.  The  Marine  squad  and  the  ANSF  fire  team  completely 
collapsed  into  the  A5  compound  and  surrounding  exterior  wall. 

After  observing  the  post-detonation  environment,  it  appeared 
appropriate  to  utilize  the  “consider  how  to  respond  immediately  to  detonation”  metric. 
As  depicted  in  Figure  14,  there  were  four  behavior  factors  related  to  this  BARS: 
awareness  of  personnel  location,  5/25  meter  searches,  establishment  and  maintaining  of  a 
cordon,  and  reporting  considerations  to  higher  headquarters.  The  transpired  events  gave 
the  impression  that  the  metrics  would  be  difficult  to  utilize.  Three  factors  made  assessing 
the  Marine  Squad  leader’s  behavior  using  the  BARS  difficult.  First,  the  affected 
casualties  required  immediate  medical  attention  so  they  were  quickly  moved  from  post¬ 
detonation  into  A5  compound  without  direct  involvement  of  the  Marine  squad  leader. 
The  inability  to  ascertain  the  Marine  squad  leader’s  verbal  commands  and  queries 
prevented  utilizing  the  “awareness  of  personnel  location”  behavior  factor.  Second,  when 
the  squad  collapsed  their  positions  into  the  A5  compound,  it  was  done  in  such  an 
expeditious  manner,  that  it  prevented  observing  whether  or  not  the  squad  conducted  a 
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5/25  meter  search  for  secondary  or  tertiary  IEDs  or  mines.  Third,  establishing  and 
sustaining  the  cordon  and  security  should  be  a  separate  and  independent  BARS.  The 
remaining  behavior  factor,  ‘reporting  considerations  to  higher  headquarters’  was 
addressed  later  in  the  scenario  training  run. 


Scenario:  Exploded  IED 

Decision:  Consider  How  to  Respond  Immediately  to  Detonation 
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Figure  14.  BARS  Version  One:  Consider  how  to  Respond  Immediately  to  Detonation. 


With  the  Marine  and  ANSF  units  moved  inside  the  A5  building 
compound,  it  was  not  possible  to  hear  radio  communication  from  the  Platoon 
Commander’s  radio  in  the  Phase  I  AAR  Room.  It  became  apparent  that  the 
communication  issue  presented  itself  again  and  the  Marine  squad  leader  and  the  Platoon 
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Commander  were  observed  trying  to  establish  2-way  communication.  This  lack  of 
communication  spanned  more  than  five  minutes  and  the  delay  adversely  affected  rating 
the  Marine  squad  leader’s  performance.  With  this  communication  delay,  it  became 
apparent  the  inclusion  of  a  time  dimension  in  any  future  BARS  would  not  be  appropriate 
given  the  difficulties  in  establishing  and  maintaining  two-way  radio  communication.  The 
BARS  for  “Consider  how  to  Respond  Immediately  to  Detonation”  was  deemed 
ineffective  for  assessing  the  Marine  squad  leader’s  decision-making  performance  for  this 
particular  cognitive  demand. 

Once  the  Platoon  Commander  (COC)  changed  his  location,  2-way 
radio  communication  was  established.  During  the  radio  transmission  from  the  Marine 
squad  leader,  the  COC  heard  the  same  improper  procedural  radio  format  (IED/UXO  9- 
Line  report).  This  made  the  Platoon  Commander  rhetorically  comment  about  there  was 
additional  training  that  was  needed  regarding  proper  radio  procedures  and  format.  In 
subsequent  two-way  radio  communication  the  Marine  squad  leader  demonstrated 
difficulty  in  medically  classifying  the  two  casualties.  After  the  radio  exchange,  the 
Platoon  Commander  commented  again  that  additional  training  was  necessary  in  medical 
classifications  and  reporting  procedures.  Finally,  the  Platoon  Commander  indicated  that 
there  was  not  a  Marine  in  the  Sweeper  role  after  the  original  Sweeper  had  become  a 
casualty.  The  Marine  squad  leader  needed  to  task  another  marine  to  that  role.  Without  a 
Marine  employed  in  the  Sweeper  role,  the  squad  had  a  very  remote  chance  of 
substantially  preventing  additional  IED/Mine  events.  The  Marines  navigated  back  to  the 
entrance  of  the  Phase  I  facility,  and  the  scenario  training  run  culminated. 

When  the  squad  established  itself  in  the  Phase  I  AAR  Room,  the 
trainer  began  the  AAR  by  covering  the  events  that  transpired  up  to  the  Sweeper  stepping 
on  the  mine.  The  trainer  elaborated  on  the  roles  and  responsibilities  of  the  Sweeper  and 
the  Marines  around  him.  IED  marking  schemes  and  techniques  were  addressed, 
specifically  shaving  cream  or  baby  powder  as  suitable  means  to  mark  IEDs  and  cleared 
areas.  Additionally,  the  trainer  elaborated  on  the  squad’s  lack  of  explicit  and/or  vocal 
lateral  communication.  The  trainer  expressed  that  only  through  vocal  and  explicit 
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communication  can  their  individual  intentions  and  movements  build  a  common 
situational  picture  for  the  other  unit  members. 

Finally,  the  trainer  paused  the  video  where  the  Sweeper  actuates 
the  mine.  The  video  shows  the  Marine  in  the  Sweeper  role  visually  recognizing  the 
presence  of  the  disturbed  earth,  but  remains  silent.  The  trainer  emphatically  proclaimed 
that  each  Marine  had  an  obligation  and  responsibility  to  individually  scan  the  area  around 
them  and  alert  the  squad  if  there  were  any  indications  of  potential  IEDs/mines.  The 
trainer  concluded  by  saying  that  it  was  imperative  for  everyone’s  survival  that  when  there 
are  confirmed  IEDs  in  the  local  area  everyone  should  remain  vigilant  in  scanning  and 
ensuring  there  are  not  secondary  and  tertiary  devices. 

Training  Day  One,  Indoor  Scenario  One — 3rd  Squad:  Due  the 

length  of  the  previous  AAR,  observation  of  this  squad’s  scenario  run  was  impossible. 
The  3rd  Squad  had  completed  their  scenario  training  run  while  the  previous  scenario  run 
debrief  was  taking  place.  In  hindsight,  the  long  occupation  of  the  AAR  room  not  only 
prevented  the  observation  of  this  scenario  run,  but  continued  long  debriefs  prevented 
subsequent  observations  as  well.  Moving  to  another  location,  such  as  the  Phase  I  video 
operator’s  room  located  between  C2  and  C4  (labeled  “out  of  play”  on  Figure  4)  could 
mitigate  this  issue. 

Interview  with  IIT  Head  Trainer 

After  explaining  the  purpose  and  intent  of  the  thesis  research,  the 
IITs  head  trainer  was  asked  to  provide  an  assessment  of  the  research  BARS  potential 
utility.  He  responded  that  BARS  would  have  to  be  strictly  an  enabler  to  the  trainer’s 
assessment.  The  head  trainer  also  commented  that  any  BARS  should  not  be  tied  to  the 
promotion  or  demotion  of  the  unit  leader  under  training.  He  stressed  that  units  under 
training  have  different  challenges  depending  on  where  they  are  in  their  PTP  cycle.  He 
explained  that  it  would  be  unfair  to  compare  a  unit’s  leadership  perfonnance  against  units 
that  are  further  along  in  their  PTP  cycle.  Sample  BARS  were  provided  for  him  to  review 
and  provide  feedback. 
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(4)  Day  Three  Observations.  The  previous  training  day  focused 
exclusively  on  squad-level  scenario  runs  in  the  indoor  (Phase  I)  facility.  Day  three 
observations  focused  exclusively  on  the  outdoor  (Phase  II)  training  range  located 
adjacent  to  the  Phase  I  facility.  Figure  15  shows  the  Phase  II  training  range  is  divided 
into  northern  and  southern  lanes  where  platoon-  and  company-level  units  conduct 
scenario  runs.  Phase  II  observations  were  exclusively  restricted  to  the  TVCS  operator’s 
room  (Figure  16),  located  in  the  IIT  administration  building  (Figure  15). 
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Figure  16.  Phase  II  Tactical  Video  Capture  System  Operator’s  Room 

(From  Thelen,  2013). 

During  the  initial  orientation  to  the  TVCS,  it  became  apparent  that 
any  observation  would  have  to  be  more  deliberate  and  focused  compared  to  Phase  I 
observations.  The  TVCS  is  comprised  of  a  large-scale  CCTV  camera  network  which  is 
remotely-controlled  and  managed  by  TVCS  operators.  The  TVCS  operator’s  console  and 
monitor  (see  Figure  16)  was  comprised  of  several  cameras  vantages  of  the  outdoor  training 
range.  Numerous  outdoor  cameras  were  selected  on  the  TVCS  operator  console  and 
monitor  and  prevented  a  singular  perspective  of  the  scenario  run.  Interactions  with  TCVS 
operators  were  limited  as  to  not  impede  their  activities  and  responsibilities.  As  a  result  of 
this  constraint,  there  was  limited  opportunity  to  control  the  camera  perspective  and  scene 
selection.  To  mitigate  this  constraint,  the  TVCS  operators  agreed  to  verbalize  the  camera 
perspectives.  Additionally,  the  TVCS  operators  agreed  to  discuss  where  the  individual 
squads  or  fire  teams  were  located  throughout  Phase  II  training  ranges.  The  two  observed 
scenario  runs  used  the  same  platoon  QRF  scenario  storyboard  (see  Figure  17). 
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Enemy  Situation:  3-6  man  enemy 
element  is  engaging  an  AHSF  unit 
conducting  a  cordon  and  search  IVO 
116  with  small  arms  fire.  The  AHSF 
unit  is  being  engaged  from  the  multiple 
positions  to  the  west,  north,  and  east 
of  the  OBJ  and  have  sustained  at  least 
(1)  friendly  casualty. 

Mission:  Provide  QRF  support  to  AMSF 
unit  in  a  SAF  engagement  in  the 
northeast  corner  of  the  village  IOT 
eliminate  all  enemy  positions  and 
assist  them  in  completing  their  cordon 
and  search  IVO  116. 

Atmospherics:  Most  civilians  are  taking 
cover  indoors  due  to  firefight. 

-_A  group  of  civilians  are  interfering 
with  cordon  positions  because  of 
civiiian  injuries  sustained  in  firefight. 

Visuals: 

-3  AMSF  teams  (5  soldiers  each)  IVO 
L5,  inside  18,  and  IV0I16  engaged  with 
enemy. 

-HME  in  116. 

-(1)  unarmed  HME  manufacturer  in 
116. 

-(2)  Wounded  AMSF  soldiers  on  the 
south  side  of  116. 

-Civilians  yelling  to  AHSF  at  the  OBJ  for 
assistance  to  treat  civilian  injuries. 
-Injured  civilians  in  H4  courtyard. 

Enemy  Activity: 

(1)  Shooter  engaging  AMSF  teams 
IVO  L5  and  in  18. 

(1)  Shooter  engaging  AMSF  team  IVO 
116  from  the  2nd  story  of  K2. 

(1)  Shooter  engaging  AHSF  team  IVO 
116  from  the  L15-L16  courtyard. 
(1)  Shooter  on  standby  IVO  H18. 

(1)  Shooter  on  standby  IVO  F18. 

(1)  Shooter  on  standby  IVO  FI. 


3/7  T-DAY  3  Scenario:  PLT  QRF 


Figure  17.  IIT  Phase  II  -  Training  Day  2,  Scenario:  Platoon  QRF  Storyboard  (From  Fennell,  2013A). 
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Outdoor  Scenario  Day  Three — 3/7  Platoon:  As  the  Marine 
platoon  entered  the  outdoor  facility,  the  platoon  split  itself  into  three  squad  elements. 
Each  element  independently  entered  through  different  entrances  along  the  northeastern 
and  western  entrances  of  the  Phase  II  north  lane  (see  Figure  17).  Additionally,  Marine 
role-players  playing  adversaries  or  ‘Shooters’  were  discharging  their  weapons  as  a  means 
to  simulate  atmospherics  of  the  scenario.  Civilian  and  ANSF  role-players  in  the  vicinity 
of  K3  compound  were  observed  requesting  the  platoon’s  assistance.  The  three  squads 
were  observed  navigating  toward  the  ANSF  and  civilian  role-players.  As  the  squads 
approached  the  vicinity  of  the  “K”  sector  of  the  Northern  training  range  (see  Figure  17, 
bold  black  letter)  they  began  to  receive  small-arms  fire  (SAF)  from  different  Shooters  in 
the  K2,  L15/16,  and  H1/H2  buildings.  Squads  subsequently  cleared  K2  and  L15/16 
buildings  of  Shooters,  but  the  H1/H2  Shooter  continued  to  engage  a  squad  in  the  vicinity 
of  K3  compound.  Finally,  the  squads  subdued  the  Shooter  in  the  vicinity  of  H1/H2 
buildings,  and  then  all  three  squads  exfiltrated  (EXFILL)  through  the  northern  exits  of  the 
Phase  II  training  range. 

During  the  platoon’s  engagement  in  the  scenario,  there  was 
considerable  difficulty  using  the  BARS.  The  inability  to  see  and/or  hear  the  Marine 
platoon  or  squad  leader’s  individual  actions  and/or  radio  communications  impeded  using 
the  BARS.  A  critical  difference  between  observing  scenario  runs  in  Phase  I  and  II  was 
the  scale  of  the  two  respective  facilities,  and  the  ability  to  focus  only  on  one  squad  vice 
three.  The  scope  of  indoor  facility  presented  a  significant  advantage  compared  to  the 
outdoor  facility  for  the  purposes  of  this  research.  As  a  result,  the  potential  exclusion  of 
the  outdoor  training  range  for  the  BARS  development  was  reevaluated  at  the  conclusion 
of  the  week’s  training  evolution.  Observation  of  the  following  platoon’s  outdoor 
scenario  run  reinforced  the  notion  of  how  difficult  rating  the  Marine  squad  leader’s 
decision  making  performance  would  be  in  the  outdoor  training  range.  Due  to  the  lack  of 
control  in  manipulating,  selecting  and  focusing  the  cameras’  point  of  view  and  display  on 
the  TVCS  monitor  and  console,  inclusion  of  the  outdoor  facility  for  this  research  was 
deemed  not  suitable. 
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d. 


Lessons  Learned  from  Second  Visit  to  ILT 


Lessons  learned  from  the  second  IIT  visit,  proved  invaluable  to  improving 
the  scope,  fonnat,  and  content  of  the  subsequent  version  of  the  BARS.  Ten  BARS 
proved  to  be  too  cumbersome  to  use  in  a  30-40  minute  fast-paced  scenario.  The  intent  of 
future  data  collection  opportunities  would  be  to  decrease  the  number  of  BARS  used. 
Additionally,  a  modification  of  the  format  and  content  of  the  BARS  metric  with  respect 
to  Part  I,  II  and  III  was  necessary. 

As  depicted  in  Figure  18,  the  BARS  Part  I  was  made  more  explicit  with 
respect  to  the  cognitive  demand  (in  accordance  with  the  MSLMM).  Additionally,  the 
Part  I  observable  decisions  were  tailored  to  a  specific  location  within  the  scenario 
training  run  (see  Figure  19).  The  decision  to  choose  a  single  scenario  run  rather  than 
multiple  scenario  runs  was  due  to  time  and  utilization  considerations  (All  squads  would 
use  training  day  one,  indoor  scenario  one).  In  Part  II,  the  number  of  proficiency  levels 
was  reduced  from  five  to  three  to  provide  more  concrete  descriptors  of  behaviors  for  the 
rating  scale.  To  reduce  ambiguity  between  proficiency  levels,  explicit  definitions  and 
descriptions  needed  to  be  placed  below  each  proficiency  level.  In  Part  III,  each  BARS 
behavior  factor  was  changed  to  focus  on  one  indicator  that  would  elicit  a  behavioral 
response.  For  example  in  Figure  18,  the  IED  emplacement  indicator  in  Part  III  is  an  ant 
trail.  The  associated  response  by  the  Marine  squad  leader  to  the  IED  emplacement 
indicator  provided  an  improved  characterization  of  the  Marine  squad  leader’s  perception 
and  decision-making  skill.  SME  interviews  elicited  observable  behavior  factors  that 
indicated  the  decision-making  proficiency  of  expert,  competent,  and  novice  decision¬ 
makers. 
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Cognitive  Task:  Notice  Indicators  of  an  IED  Device 


Decision:  Assess  Primary  IED  Emplacement  Indicator  <g>  3B 


o 


Novice 


-Does  Not  Understand 

-  Ineffective  Application 

Demonstrates  Smrted 
abdrty  to  identify  IED 
|~  ~  1  emplacement  cues  for 
thefoAovring  IED  type: 


1 


Competent 


-  Understands 

-  Effective  Application 

Demonstrates  abAty  to 

_  identify  I  ED  emplacement 

I  I  cues  for  the  foAowing  IED 
type: 


2 


Expert 


-  Thorough  Understanding 

-  Highly  Effective  Application 

Demonstrates  Adv. 
abdrty  to  identify  IED 
□  emplacement  cues  for 
the  foAowing  IED  type: 


Command-Detonated  Indicator: 


-Ant  trails 


NOTES: 


Figure  18.  BARS  Version  2  -  Assess  Primary  IED  Emplacement  Indicator  @  3B. 


Figure  19.  QRF  To  ANSF  IED  Find  with  Final  IIT  Data  Collection  BARS  Assessment 

Locations  (After  Fennell,  2013B). 
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5.  Subject  Matter  Expert  Interviews 

SME  interviews  were  crucial  to  the  development  of  the  IIT  decision-making 
performance  metrics.  The  purpose  of  the  interviews  was  to  elicit  indicators  of  decision¬ 
making  behavior  linked  to  BARS  stratification  levels  (novice,  competent,  and  expert). 
The  format  of  the  interview  script  afforded  a  conversational  approach  with  the  SMEs. 
The  interview’s  two-way  dialog  allowed  the  SMEs  to  be  actively  engaged  in  both 
direction  and  depth  of  the  topics  presented 

Each  interview  session  was  expected  to  take  approximately  two  hours  to 
complete.  Development  of  the  metrics  required  multiple  salient  perspectives  to  define 
the  descriptors  included  in  the  BARS.  Analysis  of  the  interviews  provided  the  necessary 
descriptors  of  an  expert  (high  proficiency),  competent  (standard  proficiency),  and  novice 
(rudimentary  proficiency)  decision-makers. 

a.  Participants 

Developing  behavioral  descriptors  for  the  decisions  included  in  BARS 
required  relevant  subject-matter  expertise  in  the  task  domain.  Collecting  this  information 
via  structured  personnel  interviews  was  used  to  elicit  the  SMEs  knowledge  about 
observable  behaviors  related  to  an  IED-defeat  task. 

Participants  in  this  research  were  to  be  active-duty  NCOs,  staff  non¬ 
commissioned  officers  (SNCOs),  or  commissioned  officers  with  relevant  ground  combat 
arms  experience.  Uniformed  service  members  with  backgrounds  or  billets  in  quick 
reaction  forces  (QRF),  special  operating  forces  (SOF),  or  as  infantry  instructors 
(Coyotes)  were  also  included.  Additionally,  SMEs  must  have  deployed  to  an 
expeditionary  environment  where  they  were  entitled  to  imminent  danger  pay  within  the 
last  forty-eight  months. 

b.  Equipment 

Interviews  were  audio  recorded  at  the  approval  of  the  participants  to  assist 
the  investigators  with  their  data  collection  and  analysis.  All  information  pertaining  to 
personally  identifiable  information  was  secured  in  a  manner  consistent  with  the  IRB 
protocol. 
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6. 


Post-subject  Matter  Expert  Interviews  Behaviorally-Anchored  Rating 
Scales  Development 


Six  SME  interviews  were  conducted  over  a  two-week  period.  These  interviews 
provided  the  attributes  necessary  to  discriminate  between  each  level  i.e.,  novice, 
competent,  and  expert  decision-making  proficiency.  Participants  were  provided  the 
training  scenario  (see  Figure  6),  and  their  roles  and  responsibilities  for  the  interview  were 
explained.  In  Part  III  of  Figure  20,  observable  behaviors  were  synthesized  from  the  SME 
interview  data  collection.  The  number  of  BARS  was  reduced  from  thirty-nine  to  six 
post-SME  interviews  BARS.  These  six  BARS  were  used  in  the  IIT  final  data  collection 
phase. 
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Decision:  Assess  Indicators  of  Danger  Areas  Between  C1/C4 


Novice 

-  Ineffective  application 

-No  Change  in  Formation 

-Doesn't  Indiv/FT  Bound  & 
Post  Security 

-Demonstrates  Lack  of 
Tactical  Movement 


Competent 
•  Effective  Application 

Danger  Area  Indicator: 

Chokepoint 

-Hesitates  Change 
Formation 

-Lacks  Dispersion 

-ConstantTactical 
Movement  speed 


Expert 

-  Highly  effective  application 

-Fluid  Formation  Change 

-Proper  Indiv/FT  Bound  & 
Satellite 

-Adjusts  Tactical  Movem ent 
Speed  effectively 


Decision:  Assess  Indicators  of  Danger  Areas  Between  A6/A7 


Novice 

Ineffective  application 


-No  Change  in  Formation 

-Doesn't  Indiv/FT  Bound  & 
Post  Security 

-Walks  or  Strolls  through 


1 

Competent 
-  Effective  Application 

Danger  Area  Indicator: 

Chokepoint 

-Hesitates  Change 
Formation 

-Lacks  Dispersion 

-No  Adjustment  in  Pace 


Expert 

-  Highly  effective  application 


-Fluid  Formation  Change 

-Proper  Indiv/FT  Bound  & 
Satellite 

-Slow  Movement  Through  DA 


Figure  20.  BARS  Version  3  -  Assess  Indicator  of  Danger  Area  between  C1/C4  and 

A6/A7. 
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B.  IIT  FINAL  DATA  COLLECTION  PHASE 

The  IIT  final  data  collection  occurred  during  the  first  two  of  the  four-day  training 
program.  All  observations  and  assessments  focused  on  training-day  one,  indoor  (Phase  I) 
scenario  one  training  storyboard  (see  Figure  19).  During  this  scenario,  six  behaviorally- 
anchored  rating  scales  (BARS)  were  linked  to  six  observational  indoor  (Phase  I) 
locations.  The  following  measures  section  elaborates  on  the  specific  BARS  used  for  data 
collection  in  this  scenario.  The  metrics  were  printed  and  laminated  on  8.5”  x  1 1”  paper. 
All  observations  and  assessments  were  conducted  in  either  the  MILESTONE  operator’s 
room  (located  in  the  IIT  administration  building)  or  the  Phase  I  AAR  Room.  Prior 
coordination  with  the  IIT  Operations  Officer  assured  that  the  same  name  scenario  run 
would  be  utilized  by  all  of  the  training  participant’s  squad  elements.  Observations  and 
assessments  were  conducted  in  a  manner  to  minimize  participants’  awareness  of  the 
researcher.  The  anticipated  number  of  scenario  runs  to  be  observed  ranged  from  four  to 
eight. 


1.  Participants 

The  research  participants  in  this  phase  are  from,  3ld  Marines,  7th  Regiment  based 
out  of  Marine  Corps  Base  29  Palms.  There  are  approximately  120  to  180  Marines 
including  support  personnel  in  a  Marine  infantry  company.  The  company  has  three 
infantry  line  platoons  (nine  squads)  and  a  supporting  weapons  platoon.  Observations 
were  focused  on  the  nine  to  twelve  Marine  squad  leaders  of  the  company. 

2.  Equipment 

The  observations  and  ratings  occurred  in  one  of  two  locations,  the  Phase  I  AAR 
Room  or  the  MILESTONE  Operator’s  room.  MILESTONE  system  is  an  improved 
instrumented  video-capturing  system  replacing  the  existing  Video  Flashlight/TVCS 
systems  used  in  the  Phase  I  and  II  respectively.  Currently,  the  Phase  I  facility  is  the  only 
training  range  in  the  Marine  Corps  that  uses  the  MILESTONE  system.  The  MILESTONE 
Operator’s  room  is  located  in  the  IIT  administration  building  (see  Figure  15). 
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3. 


Measures 


Six  BARS  were  utilized  in  this  phase.  Figure  19  depicted  the  approximate 
location  where  the  observations  were  focused  occurred.  The  BARS  results  and 
discussion  are  described  in  the  following  results  chapter  of  this  thesis. 
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V.  RESULTS 


This  chapter  presents  results  from  the  infantry  immersion  trainer  (IIT)  final  data 
collection  phase. 

A.  MEASURES 

The  first  four  BARS  mentioned  below  were  considered  location-specific 
and  are  depicted  in  white  text  boxes  in  the  Figure  19.  Additionally,  the  final  two  BARS 
mentioned  below  were  considered  event  specific  and  are  represented  by  yellow  text 
boxes  in  the  storyboard.  Location-specific  BARS  were  pertinent  only  to  specific  areas  or 
locations  in  the  scenario,  whereas  event-specific  BARS  were  only  applicable  when 
simulated  improvised  explosive  devices  (IEDs)  were  detonated  in  the  scenario. 

As  the  squad  navigated  from  the  southern  entrance  of  the  indoor  (Phase  I)  facility 
they  encountered  a  chokepoint  danger  area  between  the  Cl  and  C4  buildings.  This 
chokepoint  was  the  first  major  decision  point  encountered  by  the  Marine  squad  leader. 
The  chokepoint  forced  the  Marine  squad  leader  to  recognize  and  tactically  respond  to  the 
characteristics  presented  in  the  danger  area,  such  as  the  channelizing  features  of  the  L- 
shaped  alleyway,  blind  corners  where  visibility  is  obstructed,  and  the  perpendicular 
alleyway  between  Cl  and  C3  buildings  (see  Figure  19). 

The  Marine  squad  leader  used  his  communication,  location  and  proximity,  and 
techniques,  tactics  and  procedure  (TTP)  execution  to  tactically  maneuver  the  squad 
through  the  C1/C4  chokepoint.  Novice,  competent,  or  expert  proficiency  ratings  were 
attributed  to  the  associated  level  of  proficiency  descriptors  listed  in  Part  I  (red  bold  font) 
of  Figure  21.  As  the  squad  encountered  this  first  designated  rating  location,  the  rater 
used  the  descriptors  under  the  rating  scale  in  Part  III  (red  bold  font)  to  determine  the 
Marine  squad  leader’s  proficiency  level  regarding  the  following  decision-making 
behaviors:  change  in  formation,  dispersion  through  the  danger  area,  and  tactical 
movement  speed.  The  rater  then  associated  the  decision  making  behavior  with  the 
decision  making  proficiency  level  reflected  by  the  behavior. 
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Decision:  Assess  Indicators  of  Danger  Areas  Between  C1/C4 
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Figure  2 1 .  Final  BARS:  Assess  Indicators  of  Danger  Areas  Between  C 1/C4  - 

Chokepoint. 


Once  the  decision  making  behaviors  were  assessed,  they  were  rated,  marked  and 
then  averaged.  When  a  Marine  squad  leader  had  a  majority  of  ratings  (Part  III  in  Figure 
21)  linked  to  a  particular  proficiency  level,  i.e.,  0,  1,  or  2  (Part  I  of  Figure  21),  that 
proficiency  category  is  awarded  (novice,  competent,  expert).  If  there  were  equal 
distributions  of  observed  behaviors  for  the  proficiency  levels,  the  lesser  rating  will  be 
awarded.  As  shown  in  Figure  22,  there  was  one  novice  and  one  competent  rating  given, 
and  the  third  behavior  was  inconclusive  or  not  observable.  Since  the  cumulative  average 
((0+1)  /  2  =  .5)  were  less  than  competent  rating  (<1),  a  novice  rating  were  awarded. 
However,  if  three  proficiency  descriptors  were  represented  equally  across  the  novice, 
competent  and  expert  categories,  the  competent  rating  would  be  awarded.  For  example, 
in  Figure  23,  there  was  one  novice,  one  competent,  and  one  expert  rating  indicated  by  the 
red  circles.  The  average  scoring  of  the  three  ratings  ((0  +  1  +  2)  /  3  =  1)  means  that  the 
Marine  squad  leader  was  rated  a  competent  decision-making  proficiency  category  for  that 
task. 
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Decision:  Assess  Indicators  of  Danger  Areas  Between  C1/C4 
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Figure  22.  Final  BARS:  Assessment  scoring  of  the  Indicators  of  Danger  Areas  between 

C1/C4  -  Chokepoint. 


Decision:  Assess  Indicators  of  Danger  Areas  Between  C1/C4 
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Figure  23.  Final  BARS:  Assessment  scoring  of  the  Indicators  of  Danger  Areas  between 

C 1/C4  -  Chokepoint. 
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As  the  squad  navigated  from  the  C1/C4  chokepoint  toward  the  bazaar,  they  likely 
maneuvered  through  the  chokepoint  between  the  A6  and  A7  buildings  (See  Figure  19). 
The  Marine  squad  leader  either  navigated  the  squad  through  the  A6/A7chokepoint  or 
decided  to  use  an  alternate  path.  Instead  of  navigating  through  the  A6/A7  chokepoint,  the 
Marine  squad  leader  may  instead  satellite  his  individual  fire  teams.  Satelliting  is  a 
tactical  procedure  where  the  Marine  squad  leader  breaks  up  the  squad  into  autonomous 
fire  team  elements.  The  BARS  shown  in  Figure  5  assumes  that  the  Marine  squad  leader 
and  some  portion  of  the  squad  navigated  through  the  A6/A7  chokepoint. 

As  shown  in  Figure  24,  the  A6/A7  and  C1/C4  chokepoints  have  identical 
descriptors  and  were  rated  in  a  similar  manner.  The  differences  between  the  two 
chokepoints  were  their  physical  characteristics  and  proximity  to  the  bazaar  objective  area. 
The  C1/C4  chokepoint  was  a  more  straight-forward  danger  area  compared  to  the  complex 
characteristics  of  the  A6/A7  chokepoint  (“L”  shaped  vs.  “T”  shaped  chokepoints).  A 
danger  area  such  as  the  C1/C4  chokepoint  allows  the  Marines  to  break-up  the  location 
into  sequential  parts.  Conversely  in  the  A6/A7  chokepoint,  the  Marines  have  to  contend 
with  two  potential  avenues  of  approach  represented  by  the  top  of  the  “T”  shape.  Finally, 
the  proximity  of  the  bazaar  to  the  A6/A7  chokepoint  creates  a  natural  draw  and  friction 
point  for  the  Marine  squad  leader  and  his  squad. 
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Decision:  Assess  Indicators  of  Danger  Areas  Between  A6/A7 
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Figure  24.  Final  BARS:  Assess  Indicators  of  Danger  Areas  between  A6/A7  - 

Chokepoint. 


The  squad’s  mission  in  this  scenario  was  to  tactically  navigate  to  the 
bazaar,  link-up  with  the  Afghan  national  security  forces  (ANSF)  located  there,  identify 
the  point  of  origin  of  any  suspected  IED(s),  establish  a  cordon  and  security  around  the 
device(s),  and  coordinate  with  explosive  ordinance  disposal  (EOD)  to  disarm  the  IED(s). 
The  BARS  focused  on  the  Marine  squad  leader’s  decision  making  in  controlling  and 
securing  the  internal  and  outer  area  of  the  cordon.  As  depicted  in  Figure  25,  the  objective 
of  the  BARS  was  to  rate  the  Marine  squad  leader’s  observable  decisions  related  to 
securing  avenues  of  approach  (AOAs),  the  IED  marking  plan,  fire  team  security 
execution,  and  leveraging  the  ANSF. 
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Decision:  Assess  Indicators  of  Suitable  Cordon 
Environment 
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Figure  25.  Final  BARS:  Assess  Indicators  of  Suitable  Cordon  Environment:  Box  Cordon 

and  Security  Surrounding  Detected  IED. 


As  depicted  in  Figure  26,  the  focus  of  the  BARS  was  the  Marine  squad  leader’s 
decision  making  ability  to  coordinate  with  EOD  to  disann  the  cordon’s  simulated  IED(s). 
The  BARS  decision-making  descriptors  included  the  EOD  link-up  plan,  IED  marking 
plan,  civilian’s  location  and  proximity  to  the  cordon,  and  employment  of  the  squad’s 
mine  sweeping  equipment. 
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Decision:  Assess  Indicators  of  Suitable  Cordon  Environment 
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Figure  26.  Final  BARS:  Assess  Indicators  of  Suitable  Cordon  Environment: 

Coordination  with  EOD  until  IED  disarmed. 


The  following  two  BARS  were  event  specific,  rather  than  location  specific  in  the 
scenario.  Event-specific  BARS  were  concerned  with  the  simulated  unexploded 
ordinance  (UXO),  related  to  the  scenario  IEDs  and  mines.  Detonations  of  the  simulated 
UXOs  were  at  the  prerogative  of  the  accompanying  trainer.  With  that  responsibility,  the 
trainer  determined  the  simulated  UXOs  effective  casualty  range  and  its  blast-related 
effects.  When  a  detonation  occurred,  the  Marine  squad  leader  had  two  immediate 
decisions.  First,  he  must  secure  the  local  area  in  order  to  mitigate  and  prevent  subsequent 
engagements.  These  UXO  detonations  have  operated  in  combination  with  other  types  of 
attacks  such  as  small  arms  fire,  or  secondary  IEDs  and  mines.  These  combination  attacks 
are  intended  to  create  as  many  casualties  as  possible. 

As  depicted  in  Figure  27,  the  Marine  squad  leader’s  second  decision  related  to  the 
inquiry  of  the  squad’s  medical  status,  timeliness  of  medical  evacuation/  casualty 
evacuation  (MEDIVAC/CASEVAC)  9-Line  report,  MEDIVAC/CASEVAC  reporting 
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fluency,  and  radio  conduct  relating  to  tone,  cadence,  clarity,  and  brevity.  The  UXO/IED 
9-Line  report  to  higher  headquarters  must  always  be  completed  after  an  UXO  detonation. 


Decision:  Assess  Indicators  of  Reactive  Post-Detonation 
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Figure  27.  Final  BARS:  Assess  Indicators  of  Reactive  Post-Detonation:  Personnel 

Medical  Status  &  Reporting  Considerations. 


After  simulated  UXO  detonations,  Marine  squad  leaders  were  expected  to 
send  two  radio  reports  higher  headquarters.  Their  first  priority  was  the 
CASEVAC/MEDIVAC  9-Line  report  (assuming  there  were  casualties),  followed  by  the 
IED/UXO  9-Line  report.  Both  standardized  reports  were  to  be  sent  independent  of  one 
another.  As  depicted  in  Figure  28,  this  BARS  was  used  to  rate  the  four  behavioral 
descriptors  of  the  Marine  squad  leader.  The  behavioral  descriptors  related  to  the 
timeliness  of  the  IED/UXO  9-Line  report,  the  completeness  of  the  IED/UXO  9-Line 
report  (including  remarks),  IED/UXO  9-Line  reporting  fluency,  and  radio  conduct  with 
respect  to  tone,  cadence,  clarity,  and  brevity. 
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Decision:  Assess  Indicators  of  Reactive  Post-Detonation 
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Figure  28.  Final  BARS:  Improvised  Explosive  Device  Classification  and  Reporting 

Considerations. 


B.  RESULTS 

Table  6  presents  the  results  of  evaluations  of  four  Marine  squad  leaders  during 
training  day  one  and  two.  A  total  of  six  scenario  run  observations  were  made  during  the 
week.  However,  two  of  the  observations  were  excluded  from  analysis  due  to  major 
modifications  that  were  made  to  the  scenario.  These  modifications  included  an  ad  hoc 
inclusion  of  small-arms  fire  (SAF)  engagements,  changing  the  objective  area  and  IED 
locations.  As  depicted  in  Table  6,  the  remaining  four  assessments  are  listed  along  the  top 
of  the  table.  The  BARS  titles  are  listed  along  the  left-hand  side  of  the  table.  The 
assessment  score  for  each  Marine  squad  leader  was  indicated  by  a  numerical  “0” 
(novice),  “1”  (competent),  or  “2”  (expert). 
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Table  6.  Number  of  BARS  Assessments. 


Unit 

BARS:  Indicator 

1st  Sqd 

2nd  Sqd 

1st  Sqd 

2nd  Sqd 

Assess  indicators  of  danger  areas  between 
C1/C4:  Chokepoint 

0 

2 

2 

2 

Assess  indicators  of  danger  areas  between 
A6/A7:  Chokepoint 

1 

N/A 

2 

N/A 

Assess  indicators  of  suitable  cordon 
environment:  Box  cordon  and  security 
surrounding  detected  IED 

0 

0 

2 

0 

Assess  indicators  of  suitable  cordon 
environment:  Coordination  with  EOD  until 
IED  disarmed 

1 

0 

2 

1 

Assess  indicators  of  reactive  post¬ 
detonation:  Personnel  medical  status  & 
reporting  considerations 

No  IED 
Detonation 

2 

No  IED 
Detonation 

0 

Assess  indicators  of  reactive  post¬ 
detonation:  IED  classification  and  reporting 
considerations 

No  IED 
Detonation 

2 

No  IED 
Detonation 

0 

An  additional  post  hoc  assessment  of  each  squads  BARS  was  conducted  by  the 
individual  IIT  trainer  who  accompanied  each  squad.  Each  of  the  four  scenario  runs  was 
accompanied  by  a  different  trainer;  this  provided  an  independent  evaluation  of  each 
squad  leader’s  BARS  scores.  Each  trainer  was  asked  to  compare  their  evaluation  of  the 
squad  leader’s  performance  with  the  BARS  assessment  and  then  asked  to  confirm  or 
disagree  with  the  results  indicated  by  each  BARS.  Results  indicate  that  the  IIT  trainers 
generally  agreed  with  the  individual  BARS  or  did  not  contradict  the  individual  metric’s 
as  presented. 

C.  DISCUSSION 

The  goal  of  this  thesis  was  to  develop,  refine  and  employ  infantry  domain  relevant 
BARS  in  order  to  examine  their  utility  for  evaluating  Marine  squad  leaders’  decision¬ 
making  performance.  During  the  final  data  collection  phase,  six  BARS  were  used  for 
rating  the  decision  making  performance  of  four  Marine  squad  leaders.  All  observations 
were  based  on  one  training  scenario.  Only  two  of  the  four  scenario  training  runs  involved 
a  reactive  detonation  of  a  simulated  IED.  This  meant  that  the  “Assess  indicators  of 
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reactive  post-detonation:  Personnel  medical  status  &  reporting  considerations”  and 
“Assess  indicators  of  reactive  post-detonation:  IED  classification  and  reporting 
considerations”  BARS  were  not  used  for  scenario  runs  that  did  not  have  a  reactive 
detonation.  Additionally,  in  two  of  the  scenario  runs,  the  Marine  squad  leader  elected  not 
to  navigate  through  the  A6/A7  chokepoint.  If  it  was  observed  that  the  Marine  squad 
leader  did  not  personally  survey  or  encounter  the  chokepoint,  the  “Assess  indicators  of 
danger  areas  between  A6/A7:  Chokepoint”  metric  was  not  employed. 

The  limitations  of  the  analysis  presented  above  not  notwithstanding,  results 
support  the  utility  of  BARS  as  a  practical  method  for  the  assessment  of  Marine  squad 
leaders’  decision-making  performance.  The  findings  presented  here  are  particularly 
encouraging  in  that  they  demonstrate  the  realistic  possibility  of  replacing  the  nonstandard 
and  subjective  methods  currently  employed  in  LVC  infantry  training  environments  such 
as  the  IIT.  By  providing  instructors  and  evaluators  the  ability  to  use  standardized  metrics 
such  as  BARS  for  small-unit  decision-making  training  scenarios,  the  opportunity  for 
standardizing  after  action  reviews  can  improve  trainee  feedback  and  remediation. 

BARS  assessment  results  were  corroborated  by  the  IIT  trainer  who  accompanied 
each  squad.  Each  of  the  four  scenario  runs  was  accompanied  by  a  different  trainer.  Each 
trainer  was  independently  shown  the  Marine  squad  leader’s  assessment  scores  (six)  and 
was  asked  to  confirm  or  disagree  with  the  metrics’  findings.  All  the  IIT  trainers  generally 
agreed  with  or  did  not  contradict  the  individual  metric’s  findings. 

1.  Lessons  Learned  from  Thesis  Research 

There  were  several  lessons  learned  from  this  thesis  research  and  are  discussed  in 
the  following: 

•  There  exists  little  consensus  in  the  BARS  literature  on  the  metric’s  fonnat  and 
structure.  Additionally,  there  was  a  lack  of  empirical  applications  of  BARS. 
Further,  reference  in  the  literature  to  assist  in  developing  BARS  for  applied 
domains  is  sparse. 

•  An  emphasis  on  prior  coordination  is  essential  when  conducting  field  research 
at  an  operational  training  range.  Limited  knowledge  of  IIT  scenarios  and 

85 


operations  hampered  assessments  and  ratings  during  the  second  IIT  visit. 
During  the  third  visit  to  the  IIT,  limited  awareness  of  the  scenario  run 
scheduling  and  order  inhibited  set  up  and  preparation  for  each  event.  The 
researcher  had  to  constantly  request  status  updates  and  confirmations  from  the 
IIT  staff  regarding  scenario  run  times  and  squad  identification. 

•  Ad  hoc  changes  by  the  IIT  staff  during  the  third  visit,  prevented  inclusion  of 
two  of  the  six  scenario  run  observations.  There  was  little  warning  of  the 
changes  prior  to  their  introduction  into  the  scenario  run.  Since  training 
participant  leadership  drove  the  training  objectives,  the  IIT  staff  made 
modifications  on  the  fly  to  reflect  their  intent  or  wishes.  The  two  excluded 
scenario  runs  included  modifications  early  into  the  scenario  runs.  The 
changes  adversely  affected  the  researcher’s  ability  rate  the  Marine  squad 
leader  and  made  using  the  BARS  irrelevant. 

•  During  the  BARS  development,  there  was  a  consistent  trial-and -error  theme  to 
the  BARS.  During  the  second  visit  to  the  IIT,  it  was  discovered  that  some 
BARS  that  were  supposed  to  be  independent  were  actually  nested  into  the 
behavioral  descriptors  of  other  BARS.  For  example,  in  the  “Consider  how  to 
Respond  Immediately  to  Detonation”  BARS  (Figure  14),  the  fourth  behavior 
factor  focused  on  assessing  the  establishment  of  a  cordon  around  an  IED.  As 
previously  mentioned,  the  “Determine  How  to  Establish  Security  and 
Cordons”  BARS  (Figure  13)  is  an  independent  BARS  and  cognitive  task. 

•  Due  to  the  restrictions  incorporated  in  the  institutional  review  board  (IRB) 
research  protocol,  personal  observations  of  the  squads  engaged  in  training  was 
not  permitted  for  data  collection.  All  observations  and  ratings  occurred  while 
watching  video  monitors  of  the  squads  engaged  in  training.  Having  the  prior 
agreements  and  coordination  with  the  training  participants  and  the  NPS  IRB 
would  have  greatly  improved  the  quality  of  the  results. 
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VI.  RECOMMENDATIONS  AND  FUTURE  WORK 


This  thesis  research  demonstrated  a  deliberate  approach  toward  developing 
behaviorally-anchored  rating  scale  (BARS)  assessments.  Behavioral  responses  assessed 
were  specific  to  Marine  small-unit  decision  making.  The  introduction,  background,  and 
literature  review  chapters  described  the  motivation  and  scope  of  the  metric  development 
process.  The  methodology  and  results  chapters  encompassed  the  empirical  application  of 
the  method.  The  following  sections  elaborate  on  the  BARS  development  and  future  work 
rec  ommendations . 

A.  BEHAVIORALLY-ANCHORED  RATING  SCALES  DEVELOPMENT 

RECOMMENDATIONS 

This  section  presents  recommendations  from  the  BARS  development  process. 
Several  impediments  were  encountered  during  the  BARS  development  process  which 
impacted  bringing  this  assessment  method  to  reality.  The  following  recommendations 
are  based  on  the  researcher’s  experience  during  the  BARS  development  process. 

•  Research  is  more  than  finding  the  answer  -  The  goal  of  research  should  not 
be  exclusively  focused  on  the  end  state  of  the  results.  More  often,  scientific 
insight  is  garnered  from  application  of  the  scientific  process  that  can 
empirically  fail  or  go  awry.  The  scientific  method  is  predicated  on  the 
replicative  process,  rather  than  the  process’s  product.  One  goal  of  this 
research  was  to  ensure  the  process  is  repeatable  in  order  to  sustain  research 
findings.  In  other  words,  the  answer  is  less  important  than  the  information 
discovered  from  the  analysis.  In  this  BARS  development  process,  there  was 
an  initial  preoccupation  with  trying  to  make  the  BARS  relevant  to  all  training 
scenarios.  The  more  appropriate  approach  was  to  make  a  single  metric 
applicable  to  a  specific  scenario  and  determine  its  relevance.  Occam’s  razor 
exemplifies  this  concept  with  “all  other  things  being  equal,  the  simpler 
explanation  is  preferable”  (Blumer,  Ehrenfeucht,  Haussler,  &  Warmuth,  1987, 
p.  377).  Scoping  the  BARS  to  a  single  training  scenario  was  a  more  effective 
approach  to  rate  decision-making  for  this  thesis. 

•  Making  lemonade  when  the  world  gives  you  lemons  -  Thesis  research  was 
not  dissimilar  to  combat,  in  that  “no  plan  of  operations  extends  with  certainty 
beyond  the  first  contact  with  the  main  hostile  force”  (Vego,  2007,  p.  IX-63). 
Reviewing  research  literature  and  incorporating  guidance  from  thesis  advisors 
assisted  the  thesis  acclimatization  process  to  a  point.  Beyond  planning,  this 
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thesis  research  required  a  creative  and  adaptable  plan  capable  of  contending 
with  several  expected  research  difficulties. 

•  During  the  BARS  development  process,  issues  arose  during  the  analysis  of 
subject-matter  expert  (SME)  interview  data.  These  issues  were  related  to 
contradictions  in  SME  responses  such  as,  when  SMEs  independently  refuted 
other  SME  assertions.  Differences  in  data  collected  to  develop  decision¬ 
making  attributes  produced  indecision  about  which  SME  responses  were  more 
valid  or  accurate.  Compounding  this  dilemma  were  additional  constraints 
related  to  limited  personal  or  professional  experience  in  the  domain  (that  is, 
the  researcher  was  a  Marine  aviator,  not  a  Marine  infantry  officer).  Without 
an  objective  reference,  literature  became  the  sole  arbiter  to  the  inconclusive 
data.  The  takeaway  from  this  was  that  opinions,  even  infonned  opinions,  are 
nothing  more  than  experience-based  heuristics.  However,  justifying 
discrepancies  between  opinions  gathered  from  SME  interviews  proved  to 
quite  challenging  and  resulted  in  unanticipated  delays  in  the  thesis  progress. 

•  Choose  your  path  carefully  -  The  NPS  student’s  biggest  educational  career 
decision  is  the  selection  of  a  thesis  topic.  Selecting  a  thesis  topic  is  usually 
based  on  personal  experiences  that  motivate  the  researcher’s  interest.  The 
overarching  intent  of  this  research  was  to  enhance  small-unit  training  and 
assessment  methods  to  facilitate  protecting  military  service  members.  The 
goal  for  this  thesis  was  to  objectively  evaluate  small-unit  decision  making 
performance  in  the  IED-defeat  domain. 

Selection  of  a  ground  combat-arms  related  thesis  came  from  experiences 
relating  to  deployments  to  Iraq  and  Afghanistan.  After  witnessing  some  of 
the  carnage  and  loss  of  life  resulting  from  improvised  explosive  devices 
(IEDs),  the  topic  was  a  means  to  confront  my  personal  war  experiences. 
Technological  advances  have  improved  the  means  to  neutralize  and  defeat 
IEDs,  but  human  ability  lags  considerably  behind.  Perception, 
interpretation,  and  recognition  of  IED  emplacement  characteristics  are  an 
enduring  Marine  training  thrust.  While  the  research  goal  was  not  meant  to 
eliminate  the  evaluator’s  perspective,  the  focus  was  to  overlay  objective 
criteria  on  the  trainer’s  assessment  in  order  to  structure  the  after-action 
review  of  training  conducted. 

B.  FUTURE  WORK 

During  the  creative  process,  contingencies  arise  which  are  never  fully  explored 
for  a  myriad  of  reasons.  These  discoveries  are  usually  shelved  for  future  work  by 
subsequent  researchers.  For  example,  with  the  limited  availability  of  SMEs,  there  were 
difficulties  in  substantiating  the  decision-making  behaviors  specific  to  the  IIT  training 
participants.  Future  research  should  develop  and  refine  the  behaviors  under  scrutiny 

through  a  controlled  study  or  experiment.  Having  the  specific  behaviors,  such  as  those 
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identified  here,  significantly  correlated  to  training  participants’  perfonnance  will  not  only 
enhance  the  quality  of  the  BARS  development  for  the  small  unit  decision  making 
training.  As  shown  in  Table  7,  the  recommendations  explore  different  avenues  where  the 
existing  thesis  research  could  be  advanced. 


Table  7.  Follow-on  Research  Recommendations. 


RECOMMENDATIONS 

DESCRIPTIONS 

1.  Behaviorally-Anchored 
Rating  Scales 
Experimentation 

This  thesis  research  developed  an  empirical  concept 
without  statistically  verifying  the  reliability  or  validity  of 
the  BARS.  The  next  phase  of  this  research  would  be  to 
design  a  study  to  evaluate  this  BARS  assessment 
method.  The  purpose  of  the  experiment  would  be  to 
determine  the  level  of  confidence  in  this  assessment 
method  compared  to  other  approaches.  Additionally,  a 
study  could  involve  multiple  individuals  as  assessors  to 
evaluate  the  reliability  of  the  results. 

2.  Behaviorally-Anchored 
Rating  Scales 

Development  in  Alternate 
Infantry  Immersion 

Trainer  locations 

This  BARS  development  process  was  only  relevant  to 
the  Marine  Corps  Base  Camp  Pendleton  (MCBCP) 
infantry  immersion  trainer  (IIT).  With  two  additional 
facilities  at  Camp  Lejeune,  NC,  and  Kaneohe  Bay,  HI, 
additional  research  could  validate  the  BARS  approach  at 
the  other  IIT  facilities.  Additional  experimentation  could 
also  assess  the  level  of  training  effectiveness  at  each 
training  range. 

3.  Validation  of  Behaviorally 
Indicators  for  Small-Unit 
Decision  Making 

Behavioral  indicators  of  decision  making  were 
synthesized  from  six  participants  in  the  SME  interviews. 
Complementary  research  could  additionally  determine 
the  full-spectrum  of  behavioral  indicators  for  a 
statistically  representable  population.  The  initial  intent 
would  be  to  make  the  indicators  valid  only  to  IIT 
facilities  and  then  expand  the  research’s  scope  to  other 
training  ranges. 

This  thesis  research  has  two  logical  extensions  to  the  current  research 


methodology  for  the  near-term  (0-15  years),  and  long-term  (>15  years)  regimes.  As 
shown  in  Table  8,  the  following  are  the  natural  projections  of  the  research  in  those 
categorical  timeframes. 
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Table  8.  Near-  and  Long-Term  Future  Work  Recommendations. 


RECOMMENDATIONS 

DESCRIPTIONS 

1.  (Near-Term)  Behaviorally- 
Anchored  Rating  Scales 
Assessment  Expansion 

The  near-term  recommendation  of  this  BARS 
assessment  methodology  is  to  have  the  IIT  trainers 
integrate  BARS  as  part  into  their  assessment  process. 
Technological  advances  and  improvements  to  the  BARS 
would  enable  the  IIT  staff  to  iterate  the  BARS  as 
scenarios  evolved  over  time.  Additionally,  the 

incoiporation  of  granular  learning  objectives  to  the 
scenario  training  run  (establishing  and  securing  cordon 
around  a  detected  IED)  would  enhance  the  critique. 

2.  (Long-Term) 

Decentralized 
Behaviorally-Anchored 
Rating  Scales  Assessment 
System 

The  long-term  recommendation  for  the  BARS 
assessment  methodology  would  be  a  technological 
incoiporation  that  permits  military  organizations  the 
ability  to  perform  organic  assessments  of  small-unit 
decision  making.  This  would  enable  Platoon 

Commanders  to  use  BARS  to  assess  their  platoon’s 
decision  making  (and  other)  training  without  the  need 
for  external  support. 
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